[jira] [Resolved] (CALCITE-4621) SemiJoinRule throws AssertionError on ANTI join
[
https://issues.apache.org/jira/browse/CALCITE-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-4621.
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/cac28e95762532538cd3d0dce9e933ef9e4d5228
> SemiJoinRule throws AssertionError on ANTI join
> ---
>
> Key: CALCITE-4621
> URL: https://issues.apache.org/jira/browse/CALCITE-4621
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Credits to [~thomas.rebele] for discovering the issue.
> When SemiJoinRule (both {{CoreRules.JOIN_TO_SEMI_JOIN}} and
> {{CoreRules.PROJECT_TO_SEMI_JOIN}}) matches an ANTI join, it fails with the
> following error:
> {noformat}
> java.lang.AssertionError: ANTI
> at org.apache.calcite.rel.rules.SemiJoinRule.perform(SemiJoinRule.java:122)
> ...
> {noformat}
> The problem is that the rule config only forbids RIGHT and FULL joins, and
> lets ANTI go through. Later when the rule is actually executed, joins with
> type ANTI cannot be handled, hence the {{AssertionError}}.
> The rule config must be adapted to forbid also ANTI joins.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4617) Wrong offset when SortJoinTransposeRule pushes a sort node with an offset
[
https://issues.apache.org/jira/browse/CALCITE-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354350#comment-17354350
]
Ruben Q L commented on CALCITE-4617:
[~sylvaincrozon] I have the impression that your original patch was in the
right track, we just required more unit tests to verify explicitly that some
missing scenarios behave correctly.
[~julianhyde], we need to remember that {{SortJoinTransposeRule}} has a limited
applicability, and there are a few conditions that must be satisfied in order
for the rule to be applied (because otherwise it might produce wrong results):
- If join is not a left or right outer, we bail out
- If sort is not a trivial order-by, and if there is any sort column that is
not part of the input where the sort is pushed, we bail out
- If sort has an offset, and if the non-preserved side of the join is not
count-preserving against the join condition, we bail out
- If the input is already sorted and we are not reducing the number of tuples,
we bail out
> Wrong offset when SortJoinTransposeRule pushes a sort node with an offset
> -
>
> Key: CALCITE-4617
> URL: https://issues.apache.org/jira/browse/CALCITE-4617
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Sylvain Crozon
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The SortJoinTransposeRule will push a sort node past a join, and then
> duplicate the sort node on top of the join. When the sort node has an offset,
> we should only apply it once, otherwise we end up skipping twice as many rows
> as we should. The sort node added on top of the join should have a null
> offset.
>
> For example the testSortJoinTranspose6 test checks that for this initial plan
> {code}
> LogicalProject(DEPTNO=[$0], EMPNO=[$2])
> LogicalSort(offset=[2], fetch=[10])
> LogicalJoin(condition=[=($0, $9)], joinType=[right])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> the SortJoinTransposeRule should convert to
> {code}
> LogicalProject(DEPTNO=[$0], EMPNO=[$2])
> LogicalSort(offset=[2], fetch=[10])
> LogicalJoin(condition=[=($0, $9)], joinType=[right])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> LogicalSort(offset=[2], fetch=[10])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> Which will result in applying the offset twice. Instead the LogicalSort on
> top of the join should just have a null offset
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4437) The Sort rel should be decorrelated even though it has fetch or limit when it is not inside a Correlate
[
https://issues.apache.org/jira/browse/CALCITE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17280837#comment-17280837
]
Ruben Q L commented on CALCITE-4437:
[~danny0405], friendly reminder, do you think it could be possible for you to
take a look at this one, please (to see if we can squeeze it into 1.27)? Thanks
> The Sort rel should be decorrelated even though it has fetch or limit when it
> is not inside a Correlate
> ---
>
> Key: CALCITE-4437
> URL: https://issues.apache.org/jira/browse/CALCITE-4437
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The fix applied for CALCITE-4206 was "too drastic" and it resulted in Sort
> with fetch/offset being impossible to decorrelate in all cases.
> CALCITE-4333 addressed this issue but only partially (when the Sort with
> fetch/offset is on top on the plan). However, this solution is insufficient,
> because any Sort with fetch/offset that is not inside a Correlate can be
> decorrelated.
> Check this test in SqlToRelConverterTest (same test as CALCITE-4333, just
> with an extra LogicalProject on top of the LogicalSort):
> {code}
> @Test void testProjectSortLimitWithCorrelateInput() {
> final String sql = ""
> + "SELECT ename||deptno FROM\n"
> + "(SELECT deptno, ename\n"
> + "FROM\n"
> + "(SELECT DISTINCT deptno FROM emp) t1,\n"
> + " LATERAL (\n"
> + "SELECT ename, sal\n"
> + "FROM emp\n"
> + "WHERE deptno = t1.deptno)\n"
> + "ORDER BY ename DESC\n"
> + "LIMIT 3)";
> sql(sql).ok();
> }
> {code}
> The current plan is:
> {noformat}
> LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
> LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
> LogicalProject(DEPTNO=[$0], ENAME=[$1])
> LogicalCorrelate(correlation=[$cor0], joinType=[inner],
> requiredColumns=[{0}])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$1], SAL=[$5])
> LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
> It can actually decorrelated as:
> {noformat}
> LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
> LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
> LogicalProject(DEPTNO=[$0], ENAME=[$1])
> LogicalJoin(condition=[=($0, $3)], joinType=[inner])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$1], SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3221) Add MergeUnion operator in Enumerable convention
[ https://issues.apache.org/jira/browse/CALCITE-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L resolved CALCITE-3221. Resolution: Fixed Done via https://github.com/apache/calcite/commit/4cd90f36c3cf9a012e34f14129a907a4ce99c6f5 Thanks everyone for the feedback, especially [~vladimirsitnikov] for his suggestions to improve the patch. > Add MergeUnion operator in Enumerable convention > > > Key: CALCITE-3221 > URL: https://issues.apache.org/jira/browse/CALCITE-3221 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.19.0 >Reporter: Stamatis Zampetakis >Assignee: Ruben Q L >Priority: Minor > Labels: pull-request-available > Fix For: 1.27.0 > > Attachments: screenshot-1.png > > Time Spent: 15h 10m > Remaining Estimate: 0h > > Currently, the union operation offered by Calcite (see > [EnumerableDefaults.union|https://github.com/apache/calcite/blob/d98856bf1a5f5c151d004b769e14bdd368a67234/linq4j/src/main/java/org/apache/calcite/linq4j/EnumerableDefaults.java#L2747]) > "breaks" the collation (if any) of its inputs. > The goal of this issue is to create a new union algorithm in Enumerable > convention (EnumerableMergeUnion) that, given the fact that its inputs are > sorted by the same collation, will return the union / union all result > respecting this collation. > Most likely the implementation of the merge join can be useful. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4492) ExceptionInInitializerError caused by ImmutableBeans
Ruben Q L created CALCITE-4492:
--
Summary: ExceptionInInitializerError caused by ImmutableBeans
Key: CALCITE-4492
URL: https://issues.apache.org/jira/browse/CALCITE-4492
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.26.0
Reporter: Ruben Q L
While testing an application that uses Calcite, the following error is observed
at start-up on a Websphere9 environment (IBM J9 VM, Java8, Ubuntu):
{noformat}
Caused by: java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.ensureError(J9VMInternals.java:141)
at
java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:130)
...
Caused by: java.lang.RuntimeException: while binding method public default
java.lang.Object
com.onwbp.org.apache.calcite.plan.RelRule$Config.as(java.lang.Class)
at
com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:285)
at
com.onwbp.org.apache.calcite.util.ImmutableBeans.access$000(ImmutableBeans.java:51)
at
com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:64)
at
com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:61)
at
com.onwbp.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
at
com.onwbp.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
at
com.onwbp.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
at
com.onwbp.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
at com.onwbp.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
at
com.onwbp.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at
com.onwbp.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
at
com.onwbp.org.apache.calcite.util.ImmutableBeans.create_(ImmutableBeans.java:91)
at
com.onwbp.org.apache.calcite.util.ImmutableBeans.create(ImmutableBeans.java:72)
at
com.onwbp.org.apache.calcite.plan.RelRule$Config.(RelRule.java:121)
... 73 more
Caused by: java.lang.IllegalAccessException:
'com.onwbp.org.apache.calcite.plan.RelRule$Config' no access to:
'com.onwbp.org.apache.calcite.plan.RelRule$Config.as:(Config,Class)Object/invokeSpecial'
at java.lang.invoke.MethodHandles$Lookup.checkAccess(MethodHandles.java:333)
at
java.lang.invoke.MethodHandles$Lookup.unreflectSpecial(MethodHandles.java:970)
at
com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:283)
... 86 more
{noformat}
It would seem there is an issue around
{{org.apache.calcite.util.ImmutableBeans}} and the reflection mechanisms that
it uses, in the context of {{RelRule#Config}} (and specifically the default
method {{RelRule$Config.as}}).
Googling a bit, I found the following Flink ticket FLINK-19820, which has a
very similar exception message. In that case it was seen with Java9 in a Debian
environment, and the problem is also around {{ImmutableBeans}} in
{{RelBuilder#Config}}, and again the issue seems generated by a default method
({{RelBuilder$Config.toBuilder}}).
The Flink ticket was closed as "Won't Fix" just because Flink does not support
Java9 (and apparently the issue did not occur with the Flink-supported Java
versions), but the underlying Calcite problem was never tackled (and Calcite is
supposed to work on Java9).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4437) The Sort rel should be decorrelated even though it has fetch or limit when it is not inside a Correlate
[
https://issues.apache.org/jira/browse/CALCITE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4437:
---
Description:
The fix applied for CALCITE-4206 was "too drastic" and it resulted in Sort with
fetch/offset being impossible to decorrelate in all cases.
CALCITE-4333 addressed this issue but only partially (when the Sort with
fetch/offset is on top on the plan). However, this solution is insufficient,
because any Sort with fetch/offset that is not inside a Correlate can be
decorrelated.
Check this test in SqlToRelConverterTest (same test as CALCITE-4333, just with
an extra LogicalProject on top of the LogicalSort):
{code}
@Test void testProjectSortLimitWithCorrelateInput() {
final String sql = ""
+ "SELECT ename||deptno FROM\n"
+ "(SELECT deptno, ename\n"
+ "FROM\n"
+ "(SELECT DISTINCT deptno FROM emp) t1,\n"
+ " LATERAL (\n"
+ "SELECT ename, sal\n"
+ "FROM emp\n"
+ "WHERE deptno = t1.deptno)\n"
+ "ORDER BY ename DESC\n"
+ "LIMIT 3)";
sql(sql).ok();
}
{code}
The current plan is:
{noformat}
LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
LogicalProject(DEPTNO=[$0], ENAME=[$1])
LogicalCorrelate(correlation=[$cor0], joinType=[inner],
requiredColumns=[{0}])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(ENAME=[$1], SAL=[$5])
LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{noformat}
It can actually be decorrelated as:
{noformat}
LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
LogicalProject(DEPTNO=[$0], ENAME=[$1])
LogicalJoin(condition=[=($0, $3)], joinType=[inner])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(ENAME=[$1], SAL=[$5], DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{noformat}
was:
The fix applied for CALCITE-4206 was "too drastic" and it resulted in Sort with
fetch/offset being impossible to decorrelate in all cases.
CALCITE-4333 addressed this issue but only partially (when the Sort with
fetch/offset is on top on the plan). However, this solution is insufficient,
because any Sort with fetch/offset that is not inside a Correlate can be
decorrelated.
Check this test in SqlToRelConverterTest (same test as CALCITE-4333, just with
an extra LogicalProject on top of the LogicalSort):
{code}
@Test void testProjectSortLimitWithCorrelateInput() {
final String sql = ""
+ "SELECT ename||deptno FROM\n"
+ "(SELECT deptno, ename\n"
+ "FROM\n"
+ "(SELECT DISTINCT deptno FROM emp) t1,\n"
+ " LATERAL (\n"
+ "SELECT ename, sal\n"
+ "FROM emp\n"
+ "WHERE deptno = t1.deptno)\n"
+ "ORDER BY ename DESC\n"
+ "LIMIT 3)";
sql(sql).ok();
}
{code}
The current plan is:
{noformat}
LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
LogicalProject(DEPTNO=[$0], ENAME=[$1])
LogicalCorrelate(correlation=[$cor0], joinType=[inner],
requiredColumns=[{0}])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(ENAME=[$1], SAL=[$5])
LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{noformat}
It can actually decorrelated as:
{noformat}
LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
LogicalProject(DEPTNO=[$0], ENAME=[$1])
LogicalJoin(condition=[=($0, $3)], joinType=[inner])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(ENAME=[$1], SAL=[$5], DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{noformat}
> The Sort rel should be decorrelated even though it has fetch or limit when it
> is not inside a Correlate
> ---
>
> Key: CALCITE-4437
> URL: https://issues.apache.org/jira/browse/CALCITE-4437
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Priority: Major
>
[jira] [Assigned] (CALCITE-4437) The Sort rel should be decorrelated even though it has fetch or limit when it is not inside a Correlate
[
https://issues.apache.org/jira/browse/CALCITE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L reassigned CALCITE-4437:
--
Assignee: Thomas Rebele (was: Ruben Q L)
> The Sort rel should be decorrelated even though it has fetch or limit when it
> is not inside a Correlate
> ---
>
> Key: CALCITE-4437
> URL: https://issues.apache.org/jira/browse/CALCITE-4437
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Assignee: Thomas Rebele
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The fix applied for CALCITE-4206 was "too drastic" and it resulted in Sort
> with fetch/offset being impossible to decorrelate in all cases.
> CALCITE-4333 addressed this issue but only partially (when the Sort with
> fetch/offset is on top on the plan). However, this solution is insufficient,
> because any Sort with fetch/offset that is not inside a Correlate can be
> decorrelated.
> Check this test in SqlToRelConverterTest (same test as CALCITE-4333, just
> with an extra LogicalProject on top of the LogicalSort):
> {code}
> @Test void testProjectSortLimitWithCorrelateInput() {
> final String sql = ""
> + "SELECT ename||deptno FROM\n"
> + "(SELECT deptno, ename\n"
> + "FROM\n"
> + "(SELECT DISTINCT deptno FROM emp) t1,\n"
> + " LATERAL (\n"
> + "SELECT ename, sal\n"
> + "FROM emp\n"
> + "WHERE deptno = t1.deptno)\n"
> + "ORDER BY ename DESC\n"
> + "LIMIT 3)";
> sql(sql).ok();
> }
> {code}
> The current plan is:
> {noformat}
> LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
> LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
> LogicalProject(DEPTNO=[$0], ENAME=[$1])
> LogicalCorrelate(correlation=[$cor0], joinType=[inner],
> requiredColumns=[{0}])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$1], SAL=[$5])
> LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
> It can actually be decorrelated as:
> {noformat}
> LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
> LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
> LogicalProject(DEPTNO=[$0], ENAME=[$1])
> LogicalJoin(condition=[=($0, $3)], joinType=[inner])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$1], SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-4437) The Sort rel should be decorrelated even though it has fetch or limit when it is not inside a Correlate
[
https://issues.apache.org/jira/browse/CALCITE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L reassigned CALCITE-4437:
--
Assignee: Ruben Q L
> The Sort rel should be decorrelated even though it has fetch or limit when it
> is not inside a Correlate
> ---
>
> Key: CALCITE-4437
> URL: https://issues.apache.org/jira/browse/CALCITE-4437
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The fix applied for CALCITE-4206 was "too drastic" and it resulted in Sort
> with fetch/offset being impossible to decorrelate in all cases.
> CALCITE-4333 addressed this issue but only partially (when the Sort with
> fetch/offset is on top on the plan). However, this solution is insufficient,
> because any Sort with fetch/offset that is not inside a Correlate can be
> decorrelated.
> Check this test in SqlToRelConverterTest (same test as CALCITE-4333, just
> with an extra LogicalProject on top of the LogicalSort):
> {code}
> @Test void testProjectSortLimitWithCorrelateInput() {
> final String sql = ""
> + "SELECT ename||deptno FROM\n"
> + "(SELECT deptno, ename\n"
> + "FROM\n"
> + "(SELECT DISTINCT deptno FROM emp) t1,\n"
> + " LATERAL (\n"
> + "SELECT ename, sal\n"
> + "FROM emp\n"
> + "WHERE deptno = t1.deptno)\n"
> + "ORDER BY ename DESC\n"
> + "LIMIT 3)";
> sql(sql).ok();
> }
> {code}
> The current plan is:
> {noformat}
> LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
> LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
> LogicalProject(DEPTNO=[$0], ENAME=[$1])
> LogicalCorrelate(correlation=[$cor0], joinType=[inner],
> requiredColumns=[{0}])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$1], SAL=[$5])
> LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
> It can actually be decorrelated as:
> {noformat}
> LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
> LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
> LogicalProject(DEPTNO=[$0], ENAME=[$1])
> LogicalJoin(condition=[=($0, $3)], joinType=[inner])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$1], SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4492) ExceptionInInitializerError caused by ImmutableBeans
[
https://issues.apache.org/jira/browse/CALCITE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282599#comment-17282599
]
Ruben Q L commented on CALCITE-4492:
Not sure [~julianhyde], but maybe WebSphere is to blame here.
WebSphere environment in this test was:
{noformat}
WebSphere Platform 9.0.0.1 ... [JAVA8 8.0.3.11]
{noformat}
I have found several issues on IBM involving {{IllegalAccessException}} around
{{java.lang.invoke.MethodHandles$Lookup.checkAccess}}
Particularly this one seems very, very similar at stack trace level:
https://www.ibm.com/support/pages/apar/IV96259
According to IBM's ticket, this got fixed on 8.0.4.10, so maybe a WebSphere
update is just the solution, and there is no issue on Calcite side. I'll try to
confirm this point.
> ExceptionInInitializerError caused by ImmutableBeans
>
>
> Key: CALCITE-4492
> URL: https://issues.apache.org/jira/browse/CALCITE-4492
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Priority: Major
>
> While testing an application that uses Calcite, the following error is
> observed at start-up on a Websphere9 environment (IBM J9 VM, Java8, Ubuntu):
> {noformat}
> Caused by: java.lang.ExceptionInInitializerError
> at java.lang.J9VMInternals.ensureError(J9VMInternals.java:141)
> at
> java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:130)
> ...
> Caused by: java.lang.RuntimeException: while binding method public default
> java.lang.Object
> com.onwbp.org.apache.calcite.plan.RelRule$Config.as(java.lang.Class)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:285)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.access$000(ImmutableBeans.java:51)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:64)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:61)
> at
> com.onwbp.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at com.onwbp.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> at
> com.onwbp.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
> at
> com.onwbp.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create_(ImmutableBeans.java:91)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create(ImmutableBeans.java:72)
> at
> com.onwbp.org.apache.calcite.plan.RelRule$Config.(RelRule.java:121)
>
> ... 73 more
> Caused by: java.lang.IllegalAccessException:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config' no access to:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config.as:(Config,Class)Object/invokeSpecial'
> at
> java.lang.invoke.MethodHandles$Lookup.checkAccess(MethodHandles.java:333)
> at
> java.lang.invoke.MethodHandles$Lookup.unreflectSpecial(MethodHandles.java:970)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:283)
> ... 86 more
> {noformat}
> It would seem there is an issue around
> {{org.apache.calcite.util.ImmutableBeans}} and the reflection mechanisms that
> it uses, in the context of {{RelRule#Config}} (and specifically the default
> method {{RelRule$Config.as}}).
> Googling a bit, I found the following Flink ticket FLINK-19820, which has a
> very similar exception message. In that case it was seen with Java9 in a
> Debian environment, and the problem is also around {{ImmutableBeans}} in
> {{RelBuilder#Config}}, and again the issue seems generated by a default
> method ({{RelBuilder$Config.toBuilder}}).
> The Flink ticket was closed as "Won't Fix" just because Flink does not
> support Java9 (and apparently the issue did not occur with the
> Flink-supported Java versions), but the underlying Calcite problem was never
> tackled (and Calcite is supposed to work on Java9).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4492) ExceptionInInitializerError caused by ImmutableBeans
[
https://issues.apache.org/jira/browse/CALCITE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282599#comment-17282599
]
Ruben Q L edited comment on CALCITE-4492 at 2/10/21, 5:48 PM:
--
Not sure [~julianhyde], but maybe WebSphere is to blame here.
WebSphere environment in this test was:
{noformat}
WebSphere Platform 9.0.0.1 ... [JAVA8 8.0.3.11]
{noformat}
I have found several issues on IBM involving {{IllegalAccessException}} around
{{java.lang.invoke.MethodHandles$Lookup.checkAccess}}
Particularly this one seems very, very similar at stack trace level:
https://www.ibm.com/support/pages/apar/IV96259
According to IBM's ticket, this got fixed on *8.0.4.10*, so maybe a WebSphere
update is just the solution, and there is no issue on Calcite side. I'll try to
confirm this point.
was (Author: rubenql):
Not sure [~julianhyde], but maybe WebSphere is to blame here.
WebSphere environment in this test was:
{noformat}
WebSphere Platform 9.0.0.1 ... [JAVA8 8.0.3.11]
{noformat}
I have found several issues on IBM involving {{IllegalAccessException}} around
{{java.lang.invoke.MethodHandles$Lookup.checkAccess}}
Particularly this one seems very, very similar at stack trace level:
https://www.ibm.com/support/pages/apar/IV96259
According to IBM's ticket, this got fixed on 8.0.4.10, so maybe a WebSphere
update is just the solution, and there is no issue on Calcite side. I'll try to
confirm this point.
> ExceptionInInitializerError caused by ImmutableBeans
>
>
> Key: CALCITE-4492
> URL: https://issues.apache.org/jira/browse/CALCITE-4492
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Priority: Major
>
> While testing an application that uses Calcite, the following error is
> observed at start-up on a Websphere9 environment (IBM J9 VM, Java8, Ubuntu):
> {noformat}
> Caused by: java.lang.ExceptionInInitializerError
> at java.lang.J9VMInternals.ensureError(J9VMInternals.java:141)
> at
> java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:130)
> ...
> Caused by: java.lang.RuntimeException: while binding method public default
> java.lang.Object
> com.onwbp.org.apache.calcite.plan.RelRule$Config.as(java.lang.Class)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:285)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.access$000(ImmutableBeans.java:51)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:64)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:61)
> at
> com.onwbp.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at com.onwbp.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> at
> com.onwbp.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
> at
> com.onwbp.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create_(ImmutableBeans.java:91)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create(ImmutableBeans.java:72)
> at
> com.onwbp.org.apache.calcite.plan.RelRule$Config.(RelRule.java:121)
>
> ... 73 more
> Caused by: java.lang.IllegalAccessException:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config' no access to:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config.as:(Config,Class)Object/invokeSpecial'
> at
> java.lang.invoke.MethodHandles$Lookup.checkAccess(MethodHandles.java:333)
> at
> java.lang.invoke.MethodHandles$Lookup.unreflectSpecial(MethodHandles.java:970)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:283)
> ... 86 more
> {noformat}
> It would seem there is an issue around
> {{org.apache.calcite.util.ImmutableBeans}} and the reflection mechanisms that
> it uses, in the context of {{RelRule#Config}} (and specifically the default
> method {{RelRule$Config.as}}).
> Googling a bit, I found the following Flink ticket FLINK-19820, which has a
> very similar exception message. In that case it was seen with Java9 in a
> Debian environment, and the problem is also around {{ImmutableBeans}} in
> {{RelBuilder#Config}}, and again the issue seems generated by a default
> method ({{RelBuilder$Config.toBuilder}}).
> The Flink ticket was closed as "Won't Fix" just because Flink does not
> support Java9 (and apparently the
[jira] [Comment Edited] (CALCITE-4492) ExceptionInInitializerError caused by ImmutableBeans
[
https://issues.apache.org/jira/browse/CALCITE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282599#comment-17282599
]
Ruben Q L edited comment on CALCITE-4492 at 2/10/21, 5:50 PM:
--
Not sure about shading's fault [~julianhyde]; however, maybe WebSphere is to
blame here.
WebSphere environment in this test was:
{noformat}
WebSphere Platform 9.0.0.1 ... [JAVA8 8.0.3.11]
{noformat}
I have found several issues on IBM involving {{IllegalAccessException}} around
{{java.lang.invoke.MethodHandles$Lookup.checkAccess}}
Particularly this one seems very, very similar at stack trace level:
https://www.ibm.com/support/pages/apar/IV96259
According to IBM's ticket, this got fixed on *8.0.4.10*, so maybe a WebSphere
update is just the solution, and there is no issue on Calcite side. I'll try to
confirm this point.
was (Author: rubenql):
Not sure [~julianhyde], but maybe WebSphere is to blame here.
WebSphere environment in this test was:
{noformat}
WebSphere Platform 9.0.0.1 ... [JAVA8 8.0.3.11]
{noformat}
I have found several issues on IBM involving {{IllegalAccessException}} around
{{java.lang.invoke.MethodHandles$Lookup.checkAccess}}
Particularly this one seems very, very similar at stack trace level:
https://www.ibm.com/support/pages/apar/IV96259
According to IBM's ticket, this got fixed on *8.0.4.10*, so maybe a WebSphere
update is just the solution, and there is no issue on Calcite side. I'll try to
confirm this point.
> ExceptionInInitializerError caused by ImmutableBeans
>
>
> Key: CALCITE-4492
> URL: https://issues.apache.org/jira/browse/CALCITE-4492
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Priority: Major
>
> While testing an application that uses Calcite, the following error is
> observed at start-up on a Websphere9 environment (IBM J9 VM, Java8, Ubuntu):
> {noformat}
> Caused by: java.lang.ExceptionInInitializerError
> at java.lang.J9VMInternals.ensureError(J9VMInternals.java:141)
> at
> java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:130)
> ...
> Caused by: java.lang.RuntimeException: while binding method public default
> java.lang.Object
> com.onwbp.org.apache.calcite.plan.RelRule$Config.as(java.lang.Class)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:285)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.access$000(ImmutableBeans.java:51)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:64)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:61)
> at
> com.onwbp.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at com.onwbp.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> at
> com.onwbp.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
> at
> com.onwbp.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create_(ImmutableBeans.java:91)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create(ImmutableBeans.java:72)
> at
> com.onwbp.org.apache.calcite.plan.RelRule$Config.(RelRule.java:121)
>
> ... 73 more
> Caused by: java.lang.IllegalAccessException:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config' no access to:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config.as:(Config,Class)Object/invokeSpecial'
> at
> java.lang.invoke.MethodHandles$Lookup.checkAccess(MethodHandles.java:333)
> at
> java.lang.invoke.MethodHandles$Lookup.unreflectSpecial(MethodHandles.java:970)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:283)
> ... 86 more
> {noformat}
> It would seem there is an issue around
> {{org.apache.calcite.util.ImmutableBeans}} and the reflection mechanisms that
> it uses, in the context of {{RelRule#Config}} (and specifically the default
> method {{RelRule$Config.as}}).
> Googling a bit, I found the following Flink ticket FLINK-19820, which has a
> very similar exception message. In that case it was seen with Java9 in a
> Debian environment, and the problem is also around {{ImmutableBeans}} in
> {{RelBuilder#Config}}, and again the issue seems generated by a default
> method ({{RelBuilder$Config.toBuilder}}).
> The Flink ticket was closed as "Won't Fix" just because Flink does not
>
[jira] [Assigned] (CALCITE-4492) ExceptionInInitializerError caused by ImmutableBeans
[
https://issues.apache.org/jira/browse/CALCITE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L reassigned CALCITE-4492:
--
Assignee: Ruben Q L
> ExceptionInInitializerError caused by ImmutableBeans
>
>
> Key: CALCITE-4492
> URL: https://issues.apache.org/jira/browse/CALCITE-4492
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
>
> While testing an application that uses Calcite, the following error is
> observed at start-up on a Websphere9 environment (IBM J9 VM, Java8, Ubuntu):
> {noformat}
> Caused by: java.lang.ExceptionInInitializerError
> at java.lang.J9VMInternals.ensureError(J9VMInternals.java:141)
> at
> java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:130)
> ...
> Caused by: java.lang.RuntimeException: while binding method public default
> java.lang.Object
> com.onwbp.org.apache.calcite.plan.RelRule$Config.as(java.lang.Class)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:285)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.access$000(ImmutableBeans.java:51)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:64)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:61)
> at
> com.onwbp.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at com.onwbp.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> at
> com.onwbp.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
> at
> com.onwbp.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create_(ImmutableBeans.java:91)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create(ImmutableBeans.java:72)
> at
> com.onwbp.org.apache.calcite.plan.RelRule$Config.(RelRule.java:121)
>
> ... 73 more
> Caused by: java.lang.IllegalAccessException:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config' no access to:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config.as:(Config,Class)Object/invokeSpecial'
> at
> java.lang.invoke.MethodHandles$Lookup.checkAccess(MethodHandles.java:333)
> at
> java.lang.invoke.MethodHandles$Lookup.unreflectSpecial(MethodHandles.java:970)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:283)
> ... 86 more
> {noformat}
> It would seem there is an issue around
> {{org.apache.calcite.util.ImmutableBeans}} and the reflection mechanisms that
> it uses, in the context of {{RelRule#Config}} (and specifically the default
> method {{RelRule$Config.as}}).
> Googling a bit, I found the following Flink ticket FLINK-19820, which has a
> very similar exception message. In that case it was seen with Java9 in a
> Debian environment, and the problem is also around {{ImmutableBeans}} in
> {{RelBuilder#Config}}, and again the issue seems generated by a default
> method ({{RelBuilder$Config.toBuilder}}).
> The Flink ticket was closed as "Won't Fix" just because Flink does not
> support Java9 (and apparently the issue did not occur with the
> Flink-supported Java versions), but the underlying Calcite problem was never
> tackled (and Calcite is supposed to work on Java9).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3221) Add a sort-merge union algorithm
[
https://issues.apache.org/jira/browse/CALCITE-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272786#comment-17272786
]
Ruben Q L commented on CALCITE-3221:
I agree with [~julianhyde]: [~vladimirsitnikov]'s suggestion is a valid
proposal for a (separate) optimization of {{EnumerableDefaults#union}}. But
this does not collide with the current ticket's purpose: a new MergeUnion
operator that keeps the collation from its (sorted) inputs.
BTW, I think the PR is in a good shape, if anybody else wants to take a final
look, please go ahead. Otherwise I plan to merge it in the coming days.
> Add a sort-merge union algorithm
>
>
> Key: CALCITE-3221
> URL: https://issues.apache.org/jira/browse/CALCITE-3221
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.19.0
>Reporter: Stamatis Zampetakis
>Assignee: Ruben Q L
>Priority: Minor
> Labels: pull-request-available
> Attachments: screenshot-1.png
>
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> Currently, the union operation offered by Calcite is based on a {{HashSet}}
> (see
> [EnumerableDefaults.union|https://github.com/apache/calcite/blob/d98856bf1a5f5c151d004b769e14bdd368a67234/linq4j/src/main/java/org/apache/calcite/linq4j/EnumerableDefaults.java#L2747])
> and necessitates reading in memory all rows before returning a single
> result.
> Apart from increased memory consumption the operator is blocking and also
> destroys the order of its inputs.
> The goal of this issue is to add a new union algorithm (EnumerableMergeUnion
> ?) exploiting the fact that the inputs are sorted which consumes less memory
> and retains the order of its inputs.
> Most likely the implementation of the merge join can be useful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4522) CPU cost of Sort should be lower if sort keys are empty
[
https://issues.apache.org/jira/browse/CALCITE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316148#comment-17316148
]
Ruben Q L commented on CALCITE-4522:
I think this change introduces a regression in EnumerableLimitSort cost
computation, specifically in the rowCount part (issue detected by a test suite
in a downstream project).
EnumerableLimitSort cost formula used to be:
{code:java}
planner.getCostFactory().makeCost(inputRowCount, cpu, 0); // cpu is the nLogM *
bytesPerRow
{code}
After this change, the first parameter (rowCount) of this formula in case of a
Sort with fetch (e.g. an EnumerableLimitSort), will not be the inputRowCount,
but {{readCount=Math.min(inCount, offsetValue + fetchValue);}} (which in
practice in most cases would just be {{offsetValue + fetchValue}}:
{code:java}
planner.getCostFactory().makeCost(offsetValue + fetchValue, cpu, 0); // cpu is
the nLogM * bytesPerRow
{code}
In my understanding this is wrong, since a Sort operator, even with fetch (such
as EnumerableLimitSort) will still need to read and process inputRowCount of
rows (even though it just needs to keep offsetValue + fetchValue rows sorted),
so I think the new formula underestimates the cost of Sort with fetch, and its
first parameter should still be inputRowCount in all cases. Should I create a
ticket to address this issue?
> CPU cost of Sort should be lower if sort keys are empty
> ---
>
> Key: CALCITE-4522
> URL: https://issues.apache.org/jira/browse/CALCITE-4522
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: hqx
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> The old method to compute the cost of sort has some problem.
> # When the RelCollation is empty, there is no need to sort, but it still
> compute the cpu cost of sort.
> # use n * log\(n) * row_byte to estimate the cpu cost may be inaccurate,
> where n means the output row count of the sort operator, and row_byte means
> the average bytes of one row .
> Instead, I give follow suggestion.
> # the cpu cost is zero if the RelCollation is empty.
> # let heap_size be min(offset + fetch, input_count), and use input_count *
> max(1, log(heap_size))* row_byte to compute the cpu cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-2000) UNNEST a collection that has a field with nested data generates an Exception
[
https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-2000.
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/3cfeba8a4d3f5e0968820ba5a6571ceda191aa7f
> UNNEST a collection that has a field with nested data generates an Exception
>
>
> Key: CALCITE-2000
> URL: https://issues.apache.org/jira/browse/CALCITE-2000
> Project: Calcite
> Issue Type: Bug
>Reporter: Luis Fernando Kauer
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> When unnesting a column that has another level of nesting it generates the
> following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> at Baz$4$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:680)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> at
> org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:1)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> ... 29 more
> {noformat}
> The problem is that Calcite uses FlatLists internally that expect items to
> implement Comparable.
> However, a column with a nested collection will have an array or a List as
> the column value, which do not implement Comparable, so it generates the
> Exception.
> To test this in JdbcTest, I created a field in Employee with a collection.
> For simplicity, I created a field called dependents in Employee class:
> {code:java}
> public static class HrSchema {
> @Override public String toString() {
> return "HrSchema";
> }
> public final Dependent[] dependents = {
> new Dependent(10, "Michael"),
> new Dependent(10, "Jane"),
> };
> public final Employee[] emps = {
> new Employee(100, 10, "Bill", 1, 1000, Arrays.asList(dependents)),
> new Employee(200, 20, "Eric", 8000, 500,
> Collections.emptyList()),
> new Employee(150, 10, "Sebastian", 7000, null,
> Collections.emptyList()),
> new Employee(110, 10, "Theodore", 11500, 250,
> Collections.emptyList()),
> };
> public final Department[] depts = {
> new Department(10, "Sales", Arrays.asList(emps[0], emps[2]),
> new Location(-122, 38)),
> new Department(30, "Marketing", Collections.emptyList(),
> new Location(0, 52)),
> new Department(40, "HR", Collections.singletonList(emps[1]), null),
> };
> public final Dependent[] locations = {
> new Dependent(10, "San Francisco"),
> new Dependent(20, "San Diego"),
> };
> public QueryableTable foo(int count) {
> return Smalls.generateStrings(count);
> }
> public TranslatableTable view(String s) {
> return Smalls.view(s);
> }
> }
> public static class Employee {
> public final int empid;
> public final int deptno;
> public final String name;
> public final float salary;
> public final Integer commission;
> public final List dependents;
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission) {
> this(empid, deptno, name, salary, commission,
> Collections.emptyList());
> }
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission, List dependents) {
> this.empid = empid;
> this.deptno = deptno;
> this.name = name;
> this.salary =
[jira] [Commented] (CALCITE-4514) [CALCITE-4514] Fine tune the merge order of two RelSets
[ https://issues.apache.org/jira/browse/CALCITE-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293507#comment-17293507 ] Ruben Q L commented on CALCITE-4514: [~botong], it seems [~julianhyde] already took care of that via https://github.com/apache/calcite/commit/d4e1eeac6e2142d978bfaf216e85aaaf096e9a8d > [CALCITE-4514] Fine tune the merge order of two RelSets > --- > > Key: CALCITE-4514 > URL: https://issues.apache.org/jira/browse/CALCITE-4514 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Minor > Labels: pull-request-available > Fix For: 1.27.0 > > Time Spent: 40m > Remaining Estimate: 0h > > When merging two relsets, we have two preferences: > 1. Merge parent relset into child relset > 2. Merge newer relset into older relset > Currently, when the two relsets are parent set of each other, we randomly > pick a merge order without checking the second condition above. For > performance reasons, we should, to avoid unnecessary churn. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4510) Weird digests for literals with some user defined types
[
https://issues.apache.org/jira/browse/CALCITE-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289785#comment-17289785
]
Ruben Q L commented on CALCITE-4510:
Could we see the issue in a unit test?
I agree the current code is not very elegant, but it would seem it "checks the
type string suffix with an 8-character string", i.e. it checks {{endsWith("NOT
NULL")}}
And then "it trims the last 9 characters from the end of the string", just
because it also trims the white-space that precedes that string: " NOT NULL";
e.g. for a digest "XYZ NOT NULL" we end up with "XYZ"; which seems fine. Is
there an issue in that? Am I missing something?
> Weird digests for literals with some user defined types
> ---
>
> Key: CALCITE-4510
> URL: https://issues.apache.org/jira/browse/CALCITE-4510
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We find weird literals for some user defined non-nullable types. Some
> investigation shows that the problem lies in the {{RexLiteral#toJavaString}}
> method.
> In particular, it checks the type string suffix with an 8-character string:
> {noformat}
> if (!fullTypeString.endsWith("NOT NULL")) {
> {noformat}
> However, it trims the last 9 characters from the end of the string:
> {noformat}
> sb.append(fullTypeString, 0, fullTypeString.length() - 9);
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4510) RexLiteral can produce wrong digest for some user defined types
[
https://issues.apache.org/jira/browse/CALCITE-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291530#comment-17291530
]
Ruben Q L commented on CALCITE-4510:
No problem [~fan_li_ya].
Thank you for this fix.
> RexLiteral can produce wrong digest for some user defined types
> ---
>
> Key: CALCITE-4510
> URL: https://issues.apache.org/jira/browse/CALCITE-4510
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We find weird literals for some user defined non-nullable types. Some
> investigation shows that the problem lies in the {{RexLiteral#toJavaString}}
> method.
> In particular, it checks the type string suffix with an 8-character string:
> {noformat}
> if (!fullTypeString.endsWith("NOT NULL")) {
> {noformat}
> However, it trims the last 9 characters from the end of the string:
> {noformat}
> sb.append(fullTypeString, 0, fullTypeString.length() - 9);
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4510) RexLiteral can produce wrong digest for some user defined types
[
https://issues.apache.org/jira/browse/CALCITE-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4510:
---
Fix Version/s: 1.27.0
> RexLiteral can produce wrong digest for some user defined types
> ---
>
> Key: CALCITE-4510
> URL: https://issues.apache.org/jira/browse/CALCITE-4510
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We find weird literals for some user defined non-nullable types. Some
> investigation shows that the problem lies in the {{RexLiteral#toJavaString}}
> method.
> In particular, it checks the type string suffix with an 8-character string:
> {noformat}
> if (!fullTypeString.endsWith("NOT NULL")) {
> {noformat}
> However, it trims the last 9 characters from the end of the string:
> {noformat}
> sb.append(fullTypeString, 0, fullTypeString.length() - 9);
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4510) Weird digests for literals with some user defined types
[
https://issues.apache.org/jira/browse/CALCITE-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291483#comment-17291483
]
Ruben Q L commented on CALCITE-4510:
Thanks [~fan_li_ya].
I have a final remark, I think the commit message (and this Jira's title) could
be a bit more precise and descriptive. I'd suggest "RexLiteral can produce
wrong digest for some user defined types", or something like that. What do you
think?
> Weird digests for literals with some user defined types
> ---
>
> Key: CALCITE-4510
> URL: https://issues.apache.org/jira/browse/CALCITE-4510
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We find weird literals for some user defined non-nullable types. Some
> investigation shows that the problem lies in the {{RexLiteral#toJavaString}}
> method.
> In particular, it checks the type string suffix with an 8-character string:
> {noformat}
> if (!fullTypeString.endsWith("NOT NULL")) {
> {noformat}
> However, it trims the last 9 characters from the end of the string:
> {noformat}
> sb.append(fullTypeString, 0, fullTypeString.length() - 9);
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4437) The Sort rel should be decorrelated even though it has fetch or limit when it is not inside a Correlate
[
https://issues.apache.org/jira/browse/CALCITE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-4437.
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/8e7d735da428f9b193e86333ebe11712340dfeda
Thanks [~thomas.rebele] for the patch and [~danny0405] for the help!
> The Sort rel should be decorrelated even though it has fetch or limit when it
> is not inside a Correlate
> ---
>
> Key: CALCITE-4437
> URL: https://issues.apache.org/jira/browse/CALCITE-4437
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Assignee: Thomas Rebele
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The fix applied for CALCITE-4206 was "too drastic" and it resulted in Sort
> with fetch/offset being impossible to decorrelate in all cases.
> CALCITE-4333 addressed this issue but only partially (when the Sort with
> fetch/offset is on top on the plan). However, this solution is insufficient,
> because any Sort with fetch/offset that is not inside a Correlate can be
> decorrelated.
> Check this test in SqlToRelConverterTest (same test as CALCITE-4333, just
> with an extra LogicalProject on top of the LogicalSort):
> {code}
> @Test void testProjectSortLimitWithCorrelateInput() {
> final String sql = ""
> + "SELECT ename||deptno FROM\n"
> + "(SELECT deptno, ename\n"
> + "FROM\n"
> + "(SELECT DISTINCT deptno FROM emp) t1,\n"
> + " LATERAL (\n"
> + "SELECT ename, sal\n"
> + "FROM emp\n"
> + "WHERE deptno = t1.deptno)\n"
> + "ORDER BY ename DESC\n"
> + "LIMIT 3)";
> sql(sql).ok();
> }
> {code}
> The current plan is:
> {noformat}
> LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
> LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
> LogicalProject(DEPTNO=[$0], ENAME=[$1])
> LogicalCorrelate(correlation=[$cor0], joinType=[inner],
> requiredColumns=[{0}])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$1], SAL=[$5])
> LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
> It can actually be decorrelated as:
> {noformat}
> LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
> LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
> LogicalProject(DEPTNO=[$0], ENAME=[$1])
> LogicalJoin(condition=[=($0, $3)], joinType=[inner])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(ENAME=[$1], SAL=[$5], DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4510) Weird digests for literals with some user defined types
[
https://issues.apache.org/jira/browse/CALCITE-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290783#comment-17290783
]
Ruben Q L commented on CALCITE-4510:
Thanks for the explanation, [~fan_li_ya].
I agree removing this hard-coded " NOT NULL" that is all around the place and
centralize it in a single constant is a good idea (plus it solves your problem
with user-defined type systems).
Searching the code, it seems the PR could go a bit further and do the
replacement in other classes too:
- RelDatatypeImpl (already covered in the initial PR)
- RexLiteral (already covered in the initial PR)
- SqlTypeUtil#equalSansNullability
- RelOptUtil#TypeDumper#accept (x2)
And even in some tests classes:
- SqlTests#getTypeString
- SqlOperatorBaseTest (x4): checkCastToApproxOkay, checkCastToStringOkay,
checkCastToScalarOkay, assertSubFunReturns
- CalciteAssert#typeString
- RexProgramTestBase#assertNode
- RexProgramTest#assertTypeAndToString
WDYT?
> Weird digests for literals with some user defined types
> ---
>
> Key: CALCITE-4510
> URL: https://issues.apache.org/jira/browse/CALCITE-4510
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We find weird literals for some user defined non-nullable types. Some
> investigation shows that the problem lies in the {{RexLiteral#toJavaString}}
> method.
> In particular, it checks the type string suffix with an 8-character string:
> {noformat}
> if (!fullTypeString.endsWith("NOT NULL")) {
> {noformat}
> However, it trims the last 9 characters from the end of the string:
> {noformat}
> sb.append(fullTypeString, 0, fullTypeString.length() - 9);
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (CALCITE-4492) ExceptionInInitializerError caused by ImmutableBeans
[
https://issues.apache.org/jira/browse/CALCITE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L closed CALCITE-4492.
--
Resolution: Not A Problem
> ExceptionInInitializerError caused by ImmutableBeans
>
>
> Key: CALCITE-4492
> URL: https://issues.apache.org/jira/browse/CALCITE-4492
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
>
> While testing an application that uses Calcite, the following error is
> observed at start-up on a Websphere9 environment (IBM J9 VM, Java8, Ubuntu):
> {noformat}
> Caused by: java.lang.ExceptionInInitializerError
> at java.lang.J9VMInternals.ensureError(J9VMInternals.java:141)
> at
> java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:130)
> ...
> Caused by: java.lang.RuntimeException: while binding method public default
> java.lang.Object
> com.onwbp.org.apache.calcite.plan.RelRule$Config.as(java.lang.Class)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:285)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.access$000(ImmutableBeans.java:51)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:64)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans$1.load(ImmutableBeans.java:61)
> at
> com.onwbp.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at
> com.onwbp.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at com.onwbp.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> at
> com.onwbp.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
> at
> com.onwbp.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create_(ImmutableBeans.java:91)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.create(ImmutableBeans.java:72)
> at
> com.onwbp.org.apache.calcite.plan.RelRule$Config.(RelRule.java:121)
>
> ... 73 more
> Caused by: java.lang.IllegalAccessException:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config' no access to:
> 'com.onwbp.org.apache.calcite.plan.RelRule$Config.as:(Config,Class)Object/invokeSpecial'
> at
> java.lang.invoke.MethodHandles$Lookup.checkAccess(MethodHandles.java:333)
> at
> java.lang.invoke.MethodHandles$Lookup.unreflectSpecial(MethodHandles.java:970)
> at
> com.onwbp.org.apache.calcite.util.ImmutableBeans.makeDef(ImmutableBeans.java:283)
> ... 86 more
> {noformat}
> It would seem there is an issue around
> {{org.apache.calcite.util.ImmutableBeans}} and the reflection mechanisms that
> it uses, in the context of {{RelRule#Config}} (and specifically the default
> method {{RelRule$Config.as}}).
> Googling a bit, I found the following Flink ticket FLINK-19820, which has a
> very similar exception message. In that case it was seen with Java9 in a
> Debian environment, and the problem is also around {{ImmutableBeans}} in
> {{RelBuilder#Config}}, and again the issue seems generated by a default
> method ({{RelBuilder$Config.toBuilder}}).
> The Flink ticket was closed as "Won't Fix" just because Flink does not
> support Java9 (and apparently the issue did not occur with the
> Flink-supported Java versions), but the underlying Calcite problem was never
> tackled (and Calcite is supposed to work on Java9).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4522) CPU cost of Sort should be lower if sort keys are empty
[
https://issues.apache.org/jira/browse/CALCITE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317748#comment-17317748
]
Ruben Q L edited comment on CALCITE-4522 at 4/9/21, 7:48 AM:
-
That is a fair question, and honestly I'm not sure about the answer
({{RelOptCostFactory}} interface does not clarify it). However, it would seem
that, at least for some operators, this parameter is calculated with a certain
formula that represents (somehow) their computational cost, e.g.
[EnumerableHashJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableHashJoin.java#L129],
[EnumerableMergeJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableMergeJoin.java#L396],
[Correlate|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/rel/core/Correlate.java#L206],
...
In any case, this is not a blocking issue; as a workaround, I can overcome it
by defining my own metadata handler for NonCumulativeCost for EnumerableSort &
EnumerableLimitSort, but I just wanted to alert about this "regression" before
the next release is built.
was (Author: rubenql):
That is a fair question, and honestly I'm not sure about the answer
({{RelOptCostFactory}} interface does not clarify it). However, it would seem
that, at least for some operators, this parameter is calculated with a certain
formula that represents (somehow) their computational cost, e.g.
[EnumerableHashJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableHashJoin.java#L129],
[EnumerableMergeJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableMergeJoin.java#L396],
[Correlate|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/rel/core/Correlate.java#L206],
...
In any case, this is not a blocking issue; as a workaround, I can overcome it
by defining my own metadada for NonCumulativeCost for EnumerableSort &
EnumerableLimitSort, but I just wanted to alert about this "regression" before
the next release is built.
> CPU cost of Sort should be lower if sort keys are empty
> ---
>
> Key: CALCITE-4522
> URL: https://issues.apache.org/jira/browse/CALCITE-4522
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: hqx
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> The old method to compute the cost of sort has some problem.
> # When the RelCollation is empty, there is no need to sort, but it still
> compute the cpu cost of sort.
> # use n * log\(n) * row_byte to estimate the cpu cost may be inaccurate,
> where n means the output row count of the sort operator, and row_byte means
> the average bytes of one row .
> Instead, I give follow suggestion.
> # the cpu cost is zero if the RelCollation is empty.
> # let heap_size be min(offset + fetch, input_count), and use input_count *
> max(1, log(heap_size))* row_byte to compute the cpu cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4522) CPU cost of Sort should be lower if sort keys are empty
[
https://issues.apache.org/jira/browse/CALCITE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317748#comment-17317748
]
Ruben Q L commented on CALCITE-4522:
That is a fair question, and honestly I'm not sure about the answer
({{RelOptCostFactory}} interface does not clarify it). However, it would seem
that, at least for some operators, this parameter is calculated with a certain
formula that represents (somehow) their computational cost, e.g.
[EnumerableHashJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableHashJoin.java#L129],
[EnumerableMergeJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableMergeJoin.java#L396],
[Correlate|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/rel/core/Correlate.java#L206],
...
> CPU cost of Sort should be lower if sort keys are empty
> ---
>
> Key: CALCITE-4522
> URL: https://issues.apache.org/jira/browse/CALCITE-4522
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: hqx
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> The old method to compute the cost of sort has some problem.
> # When the RelCollation is empty, there is no need to sort, but it still
> compute the cpu cost of sort.
> # use n * log\(n) * row_byte to estimate the cpu cost may be inaccurate,
> where n means the output row count of the sort operator, and row_byte means
> the average bytes of one row .
> Instead, I give follow suggestion.
> # the cpu cost is zero if the RelCollation is empty.
> # let heap_size be min(offset + fetch, input_count), and use input_count *
> max(1, log(heap_size))* row_byte to compute the cpu cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4522) CPU cost of Sort should be lower if sort keys are empty
[
https://issues.apache.org/jira/browse/CALCITE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317748#comment-17317748
]
Ruben Q L edited comment on CALCITE-4522 at 4/9/21, 7:47 AM:
-
That is a fair question, and honestly I'm not sure about the answer
({{RelOptCostFactory}} interface does not clarify it). However, it would seem
that, at least for some operators, this parameter is calculated with a certain
formula that represents (somehow) their computational cost, e.g.
[EnumerableHashJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableHashJoin.java#L129],
[EnumerableMergeJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableMergeJoin.java#L396],
[Correlate|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/rel/core/Correlate.java#L206],
...
In any case, this is not a blocking issue; as a workaround, I can overcome it
by defining my own metadada for NonCumulativeCost for EnumerableSort &
EnumerableLimitSort, but I just wanted to alert about this "regression" before
the next release is built.
was (Author: rubenql):
That is a fair question, and honestly I'm not sure about the answer
({{RelOptCostFactory}} interface does not clarify it). However, it would seem
that, at least for some operators, this parameter is calculated with a certain
formula that represents (somehow) their computational cost, e.g.
[EnumerableHashJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableHashJoin.java#L129],
[EnumerableMergeJoin|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableMergeJoin.java#L396],
[Correlate|https://github.com/apache/calcite/blob/8581f0a3fe9a4f079cb4d36f02121ae22118714c/core/src/main/java/org/apache/calcite/rel/core/Correlate.java#L206],
...
> CPU cost of Sort should be lower if sort keys are empty
> ---
>
> Key: CALCITE-4522
> URL: https://issues.apache.org/jira/browse/CALCITE-4522
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: hqx
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> The old method to compute the cost of sort has some problem.
> # When the RelCollation is empty, there is no need to sort, but it still
> compute the cpu cost of sort.
> # use n * log\(n) * row_byte to estimate the cpu cost may be inaccurate,
> where n means the output row count of the sort operator, and row_byte means
> the average bytes of one row .
> Instead, I give follow suggestion.
> # the cpu cost is zero if the RelCollation is empty.
> # let heap_size be min(offset + fetch, input_count), and use input_count *
> max(1, log(heap_size))* row_byte to compute the cpu cost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4522) CPU cost of Sort should be lower if sort keys are empty
[ https://issues.apache.org/jira/browse/CALCITE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317871#comment-17317871 ] Ruben Q L commented on CALCITE-4522: Thanks for the info [~vladimirsitnikov]. I agree that some of the current cost computation formulas are questionable in some cases (and directly wrong in others, like EnumerableHashJoin where it favors joins with smaller LHR, and it should actually be the opposite). As a said, since cost is pluggable this is not a blocking issue. In any case, it seems this topic is clearly larger than the current ticket scope, so I guess we can leave this ticket as it this, and move the discussion to another thread / ticket. > CPU cost of Sort should be lower if sort keys are empty > --- > > Key: CALCITE-4522 > URL: https://issues.apache.org/jira/browse/CALCITE-4522 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: hqx >Priority: Minor > Labels: pull-request-available > Fix For: 1.27.0 > > Time Spent: 9h 50m > Remaining Estimate: 0h > > The old method to compute the cost of sort has some problem. > # When the RelCollation is empty, there is no need to sort, but it still > compute the cpu cost of sort. > # use n * log\(n) * row_byte to estimate the cpu cost may be inaccurate, > where n means the output row count of the sort operator, and row_byte means > the average bytes of one row . > Instead, I give follow suggestion. > # the cpu cost is zero if the RelCollation is empty. > # let heap_size be min(offset + fetch, input_count), and use input_count * > max(1, log(heap_size))* row_byte to compute the cpu cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4147) Rename "master" branch to "main"
[ https://issues.apache.org/jira/browse/CALCITE-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285841#comment-17285841 ] Ruben Q L commented on CALCITE-4147: Just for the record, there were several people in the community who expressed their reluctance about this change in the discussion in the [email thread|https://lists.apache.org/thread.html/r5bd747f5a634b4e07573d64fafb8ea7085a671eec86ef8fd03fc29bf%40%3Cdev.calcite.apache.org%3E]; some even cast a -1. > Rename "master" branch to "main" > > > Key: CALCITE-4147 > URL: https://issues.apache.org/jira/browse/CALCITE-4147 > Project: Calcite > Issue Type: Improvement >Reporter: Michael Mior >Assignee: Michael Mior >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Settings to change: > * GIT_COMMIT in [Calcite-Snapshots Jenkins > pipeline|https://builds.apache.org/view/A-D/view/Calcite/job/Calcite-Snapshots/configure] > * Rename [Calcite-Master Jenkins > pipeline|https://builds.apache.org/view/A-D/view/Calcite/job/Calcite-Master/configure] > * Default branch on GitHub (open INFRA ticket) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2000) UNNEST a collection that has a field with nested data generates an Exception
[
https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287151#comment-17287151
]
Ruben Q L commented on CALCITE-2000:
I have been able to provide a fix for this issue, I admit it is a bit hacky,
but I do not see any other way to solve this situation.
Basically the idea is using in {{SqlFunctions$ProductComparableListEnumerator}}
an {{Object[]}} instead of {{E[]}}. Even if we need to return a
{{FlatLists.ComparableList}} we can exploit the fact that indeed that (from
ComparableList javadoc)
{noformat}
You can create an instance whose type does not extend {@link Comparable}, but
you will get a {@link ClassCastException} at runtime when you call {@link
#compareTo(Object)} if the elements of the list do not implement {@code
Comparable}."
{noformat}
This is precisely the case of the scenario presented here (or in CALCITE-4064),
where we will have an item (array or list) that is not a Comparable. But in
these cases it is not a big problem, since we just need to create the UNNEST
result, we do not require to compare it. So with the change proposed in the PR,
we are avoiding the {{ArrayStoreException}} in the best case (e.g. in new unit
test {{simpleUnnestArrayOfRows4}}), or delaying the problem until {{compareTo}}
execution, in the worst case (if in fact a comparison-scenario arrives).
BTW, a similar Comparable-to-Object change was required in
{{SqlFunction#flatList}} introduced in 1.24 via CALCITE-4063, since we can also
reach the same situation were the item is not a Comparable, but e.g. an array
(see new unit test {{simpleUnnestArrayOfRows5}}).
> UNNEST a collection that has a field with nested data generates an Exception
>
>
> Key: CALCITE-2000
> URL: https://issues.apache.org/jira/browse/CALCITE-2000
> Project: Calcite
> Issue Type: Bug
>Reporter: Luis Fernando Kauer
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When unnesting a column that has another level of nesting it generates the
> following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> at Baz$4$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:680)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> at
> org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:1)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> ... 29 more
> {noformat}
> The problem is that Calcite uses FlatLists internally that expect items to
> implement Comparable.
> However, a column with a nested collection will have an array or a List as
> the column value, which do not implement Comparable, so it generates the
> Exception.
> To test this in JdbcTest, I created a field in Employee with a collection.
> For simplicity, I created a field called dependents in Employee class:
> {code:java}
> public static class HrSchema {
> @Override public String toString() {
> return "HrSchema";
> }
> public final Dependent[] dependents = {
> new Dependent(10, "Michael"),
> new Dependent(10, "Jane"),
> };
> public final Employee[] emps = {
> new Employee(100, 10, "Bill", 1, 1000, Arrays.asList(dependents)),
> new Employee(200, 20, "Eric", 8000, 500,
> Collections.emptyList()),
> new Employee(150, 10, "Sebastian", 7000, null,
> Collections.emptyList()),
> new Employee(110,
[jira] [Comment Edited] (CALCITE-2000) UNNEST a collection that has a field with nested data generates an Exception
[
https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287151#comment-17287151
]
Ruben Q L edited comment on CALCITE-2000 at 2/19/21, 4:12 PM:
--
I have been able to provide a fix for this issue, I admit it is a bit hacky,
but I do not see any other way to solve this situation.
Basically the idea is using in {{SqlFunctions$ProductComparableListEnumerator}}
an {{Object[]}} instead of {{E[]}}. Even if we need to return a
{{FlatLists.ComparableList}} we can exploit the fact that indeed (see
ComparableList javadoc):
{noformat}
You can create an instance whose type does not extend {@link Comparable}, but
you will get a
{@link ClassCastException} at runtime when you call
{@link #compareTo(Object)} if the elements of the list do not implement
{@code Comparable}.
{noformat}
This is precisely the case of the scenario presented here (or in CALCITE-4064),
where we will have an item (array or list) that is not a Comparable. But in
these cases, it is not a big problem, since we just need to create the UNNEST
result, we do not require to compare it. So with the change proposed in the PR,
we are avoiding the {{ArrayStoreException}} in the best case (e.g. in new unit
test {{simpleUnnestArrayOfRows4}}), or delaying the problem until {{compareTo}}
execution, in the worst case (if in fact a comparison-scenario arrives).
BTW, a similar Comparable-to-Object change was required in
{{SqlFunction#flatList}} introduced in 1.24 via CALCITE-4063, since we can also
reach the same situation were the item is not a Comparable, but e.g. an array
(see new unit test {{simpleUnnestArrayOfRows5}}).
was (Author: rubenql):
I have been able to provide a fix for this issue, I admit it is a bit hacky,
but I do not see any other way to solve this situation.
Basically the idea is using in {{SqlFunctions$ProductComparableListEnumerator}}
an {{Object[]}} instead of {{E[]}}. Even if we need to return a
{{FlatLists.ComparableList}} we can exploit the fact that indeed that (from
ComparableList javadoc)
{noformat}
You can create an instance whose type does not extend {@link Comparable}, but
you will get a {@link ClassCastException} at runtime when you call {@link
#compareTo(Object)} if the elements of the list do not implement {@code
Comparable}."
{noformat}
This is precisely the case of the scenario presented here (or in CALCITE-4064),
where we will have an item (array or list) that is not a Comparable. But in
these cases it is not a big problem, since we just need to create the UNNEST
result, we do not require to compare it. So with the change proposed in the PR,
we are avoiding the {{ArrayStoreException}} in the best case (e.g. in new unit
test {{simpleUnnestArrayOfRows4}}), or delaying the problem until {{compareTo}}
execution, in the worst case (if in fact a comparison-scenario arrives).
BTW, a similar Comparable-to-Object change was required in
{{SqlFunction#flatList}} introduced in 1.24 via CALCITE-4063, since we can also
reach the same situation were the item is not a Comparable, but e.g. an array
(see new unit test {{simpleUnnestArrayOfRows5}}).
> UNNEST a collection that has a field with nested data generates an Exception
>
>
> Key: CALCITE-2000
> URL: https://issues.apache.org/jira/browse/CALCITE-2000
> Project: Calcite
> Issue Type: Bug
>Reporter: Luis Fernando Kauer
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When unnesting a column that has another level of nesting it generates the
> following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> at Baz$4$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:680)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> at
> org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> at
>
[jira] [Comment Edited] (CALCITE-2000) UNNEST a collection that has a field with nested data generates an Exception
[
https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287182#comment-17287182
]
Ruben Q L edited comment on CALCITE-2000 at 2/19/21, 5:06 PM:
--
[~julianhyde], we discussed briefly in CALCITE-4064 (which I closed as a
duplicate of this one). I'd appreciate if you could take a look at the fix I
propose here.
was (Author: rubenql):
[~julianhyde], we discussed briefly in CALCITE-4064 (which I closed as a
duplicate of this one). I'd appreciate if you could take a look a look at the
fix I propose here.
> UNNEST a collection that has a field with nested data generates an Exception
>
>
> Key: CALCITE-2000
> URL: https://issues.apache.org/jira/browse/CALCITE-2000
> Project: Calcite
> Issue Type: Bug
>Reporter: Luis Fernando Kauer
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When unnesting a column that has another level of nesting it generates the
> following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> at Baz$4$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:680)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> at
> org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:1)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> ... 29 more
> {noformat}
> The problem is that Calcite uses FlatLists internally that expect items to
> implement Comparable.
> However, a column with a nested collection will have an array or a List as
> the column value, which do not implement Comparable, so it generates the
> Exception.
> To test this in JdbcTest, I created a field in Employee with a collection.
> For simplicity, I created a field called dependents in Employee class:
> {code:java}
> public static class HrSchema {
> @Override public String toString() {
> return "HrSchema";
> }
> public final Dependent[] dependents = {
> new Dependent(10, "Michael"),
> new Dependent(10, "Jane"),
> };
> public final Employee[] emps = {
> new Employee(100, 10, "Bill", 1, 1000, Arrays.asList(dependents)),
> new Employee(200, 20, "Eric", 8000, 500,
> Collections.emptyList()),
> new Employee(150, 10, "Sebastian", 7000, null,
> Collections.emptyList()),
> new Employee(110, 10, "Theodore", 11500, 250,
> Collections.emptyList()),
> };
> public final Department[] depts = {
> new Department(10, "Sales", Arrays.asList(emps[0], emps[2]),
> new Location(-122, 38)),
> new Department(30, "Marketing", Collections.emptyList(),
> new Location(0, 52)),
> new Department(40, "HR", Collections.singletonList(emps[1]), null),
> };
> public final Dependent[] locations = {
> new Dependent(10, "San Francisco"),
> new Dependent(20, "San Diego"),
> };
> public QueryableTable foo(int count) {
> return Smalls.generateStrings(count);
> }
> public TranslatableTable view(String s) {
> return Smalls.view(s);
> }
> }
> public static class Employee {
> public final int empid;
> public final int deptno;
> public final String name;
> public final float salary;
> public final Integer commission;
> public final List dependents;
> public Employee(int empid, int deptno, String name, float salary,
>
[jira] [Updated] (CALCITE-2000) UNNEST a collection that has a field with nested data generates an Exception
[
https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-2000:
---
Fix Version/s: 1.27.0
> UNNEST a collection that has a field with nested data generates an Exception
>
>
> Key: CALCITE-2000
> URL: https://issues.apache.org/jira/browse/CALCITE-2000
> Project: Calcite
> Issue Type: Bug
>Reporter: Luis Fernando Kauer
>Assignee: Ruben Q L
>Priority: Major
> Fix For: 1.27.0
>
>
> When unnesting a column that has another level of nesting it generates the
> following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> at Baz$4$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:680)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> at
> org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:1)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> ... 29 more
> {noformat}
> The problem is that Calcite uses FlatLists internally that expect items to
> implement Comparable.
> However, a column with a nested collection will have an array or a List as
> the column value, which do not implement Comparable, so it generates the
> Exception.
> To test this in JdbcTest, I created a field in Employee with a collection.
> For simplicity, I created a field called dependents in Employee class:
> {code:java}
> public static class HrSchema {
> @Override public String toString() {
> return "HrSchema";
> }
> public final Dependent[] dependents = {
> new Dependent(10, "Michael"),
> new Dependent(10, "Jane"),
> };
> public final Employee[] emps = {
> new Employee(100, 10, "Bill", 1, 1000, Arrays.asList(dependents)),
> new Employee(200, 20, "Eric", 8000, 500,
> Collections.emptyList()),
> new Employee(150, 10, "Sebastian", 7000, null,
> Collections.emptyList()),
> new Employee(110, 10, "Theodore", 11500, 250,
> Collections.emptyList()),
> };
> public final Department[] depts = {
> new Department(10, "Sales", Arrays.asList(emps[0], emps[2]),
> new Location(-122, 38)),
> new Department(30, "Marketing", Collections.emptyList(),
> new Location(0, 52)),
> new Department(40, "HR", Collections.singletonList(emps[1]), null),
> };
> public final Dependent[] locations = {
> new Dependent(10, "San Francisco"),
> new Dependent(20, "San Diego"),
> };
> public QueryableTable foo(int count) {
> return Smalls.generateStrings(count);
> }
> public TranslatableTable view(String s) {
> return Smalls.view(s);
> }
> }
> public static class Employee {
> public final int empid;
> public final int deptno;
> public final String name;
> public final float salary;
> public final Integer commission;
> public final List dependents;
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission) {
> this(empid, deptno, name, salary, commission,
> Collections.emptyList());
> }
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission, List dependents) {
> this.empid = empid;
> this.deptno = deptno;
> this.name = name;
> this.salary = salary;
> this.commission = commission;
> this.dependents = dependents;
> }
> @Override public String toString() {
> return "Employee [empid: " + empid + ", deptno: " +
[jira] [Comment Edited] (CALCITE-2000) UNNEST a collection that has a field with nested data generates an Exception
[
https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287151#comment-17287151
]
Ruben Q L edited comment on CALCITE-2000 at 2/19/21, 4:13 PM:
--
I have been able to provide a fix for this issue, I admit it is a bit hacky,
but I do not see any other way to solve this situation.
Basically the idea is using in {{SqlFunctions$ProductComparableListEnumerator}}
an {{Object[]}} instead of {{E[]}}. Even if we need to return a
{{FlatLists.ComparableList}} we can exploit the fact that indeed (see
ComparableList javadoc):
{noformat}
You can create an instance whose type
parameter {@code T} does not extend {@link Comparable}, but you will get a
{@link ClassCastException} at runtime when you call
{@link #compareTo(Object)} if the elements of the list do not implement
{@code Comparable}.
{noformat}
This is precisely the case of the scenario presented here (or in CALCITE-4064),
where we will have an item (array or list) that is not a Comparable. But in
these cases, it is not a big problem, since we just need to create the UNNEST
result, we do not require to compare it. So with the change proposed in the PR,
we are avoiding the {{ArrayStoreException}} in the best case (e.g. in new unit
test {{simpleUnnestArrayOfRows4}}), or delaying the problem until {{compareTo}}
execution, in the worst case (if in fact a comparison-scenario arrives).
BTW, a similar Comparable-to-Object change was required in
{{SqlFunction#flatList}} introduced in 1.24 via CALCITE-4063, since we can also
reach the same situation were the item is not a Comparable, but e.g. an array
(see new unit test {{simpleUnnestArrayOfRows5}}).
was (Author: rubenql):
I have been able to provide a fix for this issue, I admit it is a bit hacky,
but I do not see any other way to solve this situation.
Basically the idea is using in {{SqlFunctions$ProductComparableListEnumerator}}
an {{Object[]}} instead of {{E[]}}. Even if we need to return a
{{FlatLists.ComparableList}} we can exploit the fact that indeed (see
ComparableList javadoc):
{noformat}
You can create an instance whose type does not extend {@link Comparable}, but
you will get a
{@link ClassCastException} at runtime when you call
{@link #compareTo(Object)} if the elements of the list do not implement
{@code Comparable}.
{noformat}
This is precisely the case of the scenario presented here (or in CALCITE-4064),
where we will have an item (array or list) that is not a Comparable. But in
these cases, it is not a big problem, since we just need to create the UNNEST
result, we do not require to compare it. So with the change proposed in the PR,
we are avoiding the {{ArrayStoreException}} in the best case (e.g. in new unit
test {{simpleUnnestArrayOfRows4}}), or delaying the problem until {{compareTo}}
execution, in the worst case (if in fact a comparison-scenario arrives).
BTW, a similar Comparable-to-Object change was required in
{{SqlFunction#flatList}} introduced in 1.24 via CALCITE-4063, since we can also
reach the same situation were the item is not a Comparable, but e.g. an array
(see new unit test {{simpleUnnestArrayOfRows5}}).
> UNNEST a collection that has a field with nested data generates an Exception
>
>
> Key: CALCITE-2000
> URL: https://issues.apache.org/jira/browse/CALCITE-2000
> Project: Calcite
> Issue Type: Bug
>Reporter: Luis Fernando Kauer
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When unnesting a column that has another level of nesting it generates the
> following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> at Baz$4$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:680)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> at
> org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> at
>
[jira] [Assigned] (CALCITE-2000) UNNEST a collection that has a field with nested data generates an Exception
[
https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L reassigned CALCITE-2000:
--
Assignee: Ruben Q L
> UNNEST a collection that has a field with nested data generates an Exception
>
>
> Key: CALCITE-2000
> URL: https://issues.apache.org/jira/browse/CALCITE-2000
> Project: Calcite
> Issue Type: Bug
>Reporter: Luis Fernando Kauer
>Assignee: Ruben Q L
>Priority: Major
>
> When unnesting a column that has another level of nesting it generates the
> following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> at Baz$4$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:680)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> at
> org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:1)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> ... 29 more
> {noformat}
> The problem is that Calcite uses FlatLists internally that expect items to
> implement Comparable.
> However, a column with a nested collection will have an array or a List as
> the column value, which do not implement Comparable, so it generates the
> Exception.
> To test this in JdbcTest, I created a field in Employee with a collection.
> For simplicity, I created a field called dependents in Employee class:
> {code:java}
> public static class HrSchema {
> @Override public String toString() {
> return "HrSchema";
> }
> public final Dependent[] dependents = {
> new Dependent(10, "Michael"),
> new Dependent(10, "Jane"),
> };
> public final Employee[] emps = {
> new Employee(100, 10, "Bill", 1, 1000, Arrays.asList(dependents)),
> new Employee(200, 20, "Eric", 8000, 500,
> Collections.emptyList()),
> new Employee(150, 10, "Sebastian", 7000, null,
> Collections.emptyList()),
> new Employee(110, 10, "Theodore", 11500, 250,
> Collections.emptyList()),
> };
> public final Department[] depts = {
> new Department(10, "Sales", Arrays.asList(emps[0], emps[2]),
> new Location(-122, 38)),
> new Department(30, "Marketing", Collections.emptyList(),
> new Location(0, 52)),
> new Department(40, "HR", Collections.singletonList(emps[1]), null),
> };
> public final Dependent[] locations = {
> new Dependent(10, "San Francisco"),
> new Dependent(20, "San Diego"),
> };
> public QueryableTable foo(int count) {
> return Smalls.generateStrings(count);
> }
> public TranslatableTable view(String s) {
> return Smalls.view(s);
> }
> }
> public static class Employee {
> public final int empid;
> public final int deptno;
> public final String name;
> public final float salary;
> public final Integer commission;
> public final List dependents;
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission) {
> this(empid, deptno, name, salary, commission,
> Collections.emptyList());
> }
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission, List dependents) {
> this.empid = empid;
> this.deptno = deptno;
> this.name = name;
> this.salary = salary;
> this.commission = commission;
> this.dependents = dependents;
> }
> @Override public String toString() {
> return "Employee [empid: " + empid + ", deptno: " + deptno
> + ",
[jira] [Commented] (CALCITE-2000) UNNEST a collection that has a field with nested data generates an Exception
[
https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287182#comment-17287182
]
Ruben Q L commented on CALCITE-2000:
[~julianhyde], we discussed briefly in CALCITE-4064 (which I closed as a
duplicate of this one). I'd appreciate if you could take a look a look at the
fix I propose here.
> UNNEST a collection that has a field with nested data generates an Exception
>
>
> Key: CALCITE-2000
> URL: https://issues.apache.org/jira/browse/CALCITE-2000
> Project: Calcite
> Issue Type: Bug
>Reporter: Luis Fernando Kauer
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When unnesting a column that has another level of nesting it generates the
> following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> at
> org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> at
> org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> at Baz$4$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:680)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> at
> org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:1)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> ... 29 more
> {noformat}
> The problem is that Calcite uses FlatLists internally that expect items to
> implement Comparable.
> However, a column with a nested collection will have an array or a List as
> the column value, which do not implement Comparable, so it generates the
> Exception.
> To test this in JdbcTest, I created a field in Employee with a collection.
> For simplicity, I created a field called dependents in Employee class:
> {code:java}
> public static class HrSchema {
> @Override public String toString() {
> return "HrSchema";
> }
> public final Dependent[] dependents = {
> new Dependent(10, "Michael"),
> new Dependent(10, "Jane"),
> };
> public final Employee[] emps = {
> new Employee(100, 10, "Bill", 1, 1000, Arrays.asList(dependents)),
> new Employee(200, 20, "Eric", 8000, 500,
> Collections.emptyList()),
> new Employee(150, 10, "Sebastian", 7000, null,
> Collections.emptyList()),
> new Employee(110, 10, "Theodore", 11500, 250,
> Collections.emptyList()),
> };
> public final Department[] depts = {
> new Department(10, "Sales", Arrays.asList(emps[0], emps[2]),
> new Location(-122, 38)),
> new Department(30, "Marketing", Collections.emptyList(),
> new Location(0, 52)),
> new Department(40, "HR", Collections.singletonList(emps[1]), null),
> };
> public final Dependent[] locations = {
> new Dependent(10, "San Francisco"),
> new Dependent(20, "San Diego"),
> };
> public QueryableTable foo(int count) {
> return Smalls.generateStrings(count);
> }
> public TranslatableTable view(String s) {
> return Smalls.view(s);
> }
> }
> public static class Employee {
> public final int empid;
> public final int deptno;
> public final String name;
> public final float salary;
> public final Integer commission;
> public final List dependents;
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission) {
> this(empid, deptno, name, salary, commission,
> Collections.emptyList());
> }
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission, List dependents) {
>
[jira] [Commented] (CALCITE-4741) AbstractRelNode#getId can overflow into a negative value causing CompileException in certain Enumerable implement methods
[
https://issues.apache.org/jira/browse/CALCITE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402078#comment-17402078
]
Ruben Q L commented on CALCITE-4741:
Thanks [~hyuan] for pointing it out! I will link that ticket to the current one.
> AbstractRelNode#getId can overflow into a negative value causing
> CompileException in certain Enumerable implement methods
> -
>
> Key: CALCITE-4741
> URL: https://issues.apache.org/jira/browse/CALCITE-4741
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{AbstractRelNode#getId}} can overflow into a negative value.
> This id is used by the {{implement}} method of several Enumerable operators
> ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to guarantee
> a unique variable name in the dynamic code, e.g.:
> {code:java}
> final ParameterExpression corrArgList =
> Expressions.parameter(Modifier.FINAL,
> List.class, "corrList" + this.getId());
> {code}
> However, if the id is negative, this variable name will have a "-", causing a
> {{CompileException}}:
> {noformat}
> ...
> final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
> org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
> left0, new org.apache.calcite.linq4j.function.Function1() {
> public org.apache.calcite.linq4j.AbstractEnumerable apply(final
> java.util.List corrList-1794407755) {
> ...
> Caused by: java.util.concurrent.ExecutionException:
> org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of
> ', )' expected instead of '-'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4741) AbstractRelNode#getId can overflow into a negative value causing CompileException in certain Enumerable implement methods
Ruben Q L created CALCITE-4741:
--
Summary: AbstractRelNode#getId can overflow into a negative value
causing CompileException in certain Enumerable implement methods
Key: CALCITE-4741
URL: https://issues.apache.org/jira/browse/CALCITE-4741
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.27.0
Reporter: Ruben Q L
Fix For: 1.28.0
{{AbstractRelNode#getId}} can overflow into a negative value.
This id is used by certain in the {{implement}} method of several Enumerable
operators ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to
guarantee a unique variable name in the dynamic code, e.g.:
{code:java}
final ParameterExpression corrArgList =
Expressions.parameter(Modifier.FINAL,
List.class, "corrList" + this.getId());
{code}
However, if the id is negative, this variable name will have a "-", causing a
{{CompileException}}:
{noformat}
...
final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
left0, new org.apache.calcite.linq4j.function.Function1() {
public org.apache.calcite.linq4j.AbstractEnumerable apply(final
java.util.List corrList-1794407755) {
{
...
Caused by: java.util.concurrent.ExecutionException:
org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of ',
)' expected instead of '-'
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4741) AbstractRelNode#getId can overflow into a negative value causing CompileException in certain Enumerable implement methods
[
https://issues.apache.org/jira/browse/CALCITE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4741:
---
Description:
{{AbstractRelNode#getId}} can overflow into a negative value.
This id is used by certain in the {{implement}} method of several Enumerable
operators ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to
guarantee a unique variable name in the dynamic code, e.g.:
{code:java}
final ParameterExpression corrArgList =
Expressions.parameter(Modifier.FINAL,
List.class, "corrList" + this.getId());
{code}
However, if the id is negative, this variable name will have a "-", causing a
{{CompileException}}:
{noformat}
...
final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
left0, new org.apache.calcite.linq4j.function.Function1() {
public org.apache.calcite.linq4j.AbstractEnumerable apply(final
java.util.List corrList-1794407755) {
...
Caused by: java.util.concurrent.ExecutionException:
org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of ',
)' expected instead of '-'
{noformat}
was:
{{AbstractRelNode#getId}} can overflow into a negative value.
This id is used by certain in the {{implement}} method of several Enumerable
operators ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to
guarantee a unique variable name in the dynamic code, e.g.:
{code:java}
final ParameterExpression corrArgList =
Expressions.parameter(Modifier.FINAL,
List.class, "corrList" + this.getId());
{code}
However, if the id is negative, this variable name will have a "-", causing a
{{CompileException}}:
{noformat}
...
final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
left0, new org.apache.calcite.linq4j.function.Function1() {
public org.apache.calcite.linq4j.AbstractEnumerable apply(final
java.util.List corrList-1794407755) {
{
...
Caused by: java.util.concurrent.ExecutionException:
org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of ',
)' expected instead of '-'
{noformat}
> AbstractRelNode#getId can overflow into a negative value causing
> CompileException in certain Enumerable implement methods
> -
>
> Key: CALCITE-4741
> URL: https://issues.apache.org/jira/browse/CALCITE-4741
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Priority: Minor
> Fix For: 1.28.0
>
>
> {{AbstractRelNode#getId}} can overflow into a negative value.
> This id is used by certain in the {{implement}} method of several Enumerable
> operators ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to
> guarantee a unique variable name in the dynamic code, e.g.:
> {code:java}
> final ParameterExpression corrArgList =
> Expressions.parameter(Modifier.FINAL,
> List.class, "corrList" + this.getId());
> {code}
> However, if the id is negative, this variable name will have a "-", causing a
> {{CompileException}}:
> {noformat}
> ...
> final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
> org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
> left0, new org.apache.calcite.linq4j.function.Function1() {
> public org.apache.calcite.linq4j.AbstractEnumerable apply(final
> java.util.List corrList-1794407755) {
> ...
> Caused by: java.util.concurrent.ExecutionException:
> org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of
> ', )' expected instead of '-'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4741) AbstractRelNode#getId can overflow into a negative value causing CompileException in certain Enumerable implement methods
[
https://issues.apache.org/jira/browse/CALCITE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4741:
---
Description:
{{AbstractRelNode#getId}} can overflow into a negative value.
This id is used by the {{implement}} method of several Enumerable operators
({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to guarantee a
unique variable name in the dynamic code, e.g.:
{code:java}
final ParameterExpression corrArgList =
Expressions.parameter(Modifier.FINAL,
List.class, "corrList" + this.getId());
{code}
However, if the id is negative, this variable name will have a "-", causing a
{{CompileException}}:
{noformat}
...
final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
left0, new org.apache.calcite.linq4j.function.Function1() {
public org.apache.calcite.linq4j.AbstractEnumerable apply(final
java.util.List corrList-1794407755) {
...
Caused by: java.util.concurrent.ExecutionException:
org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of ',
)' expected instead of '-'
{noformat}
was:
{{AbstractRelNode#getId}} can overflow into a negative value.
This id is used by certain in the {{implement}} method of several Enumerable
operators ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to
guarantee a unique variable name in the dynamic code, e.g.:
{code:java}
final ParameterExpression corrArgList =
Expressions.parameter(Modifier.FINAL,
List.class, "corrList" + this.getId());
{code}
However, if the id is negative, this variable name will have a "-", causing a
{{CompileException}}:
{noformat}
...
final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
left0, new org.apache.calcite.linq4j.function.Function1() {
public org.apache.calcite.linq4j.AbstractEnumerable apply(final
java.util.List corrList-1794407755) {
...
Caused by: java.util.concurrent.ExecutionException:
org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of ',
)' expected instead of '-'
{noformat}
> AbstractRelNode#getId can overflow into a negative value causing
> CompileException in certain Enumerable implement methods
> -
>
> Key: CALCITE-4741
> URL: https://issues.apache.org/jira/browse/CALCITE-4741
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Priority: Minor
> Fix For: 1.28.0
>
>
> {{AbstractRelNode#getId}} can overflow into a negative value.
> This id is used by the {{implement}} method of several Enumerable operators
> ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to guarantee
> a unique variable name in the dynamic code, e.g.:
> {code:java}
> final ParameterExpression corrArgList =
> Expressions.parameter(Modifier.FINAL,
> List.class, "corrList" + this.getId());
> {code}
> However, if the id is negative, this variable name will have a "-", causing a
> {{CompileException}}:
> {noformat}
> ...
> final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
> org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
> left0, new org.apache.calcite.linq4j.function.Function1() {
> public org.apache.calcite.linq4j.AbstractEnumerable apply(final
> java.util.List corrList-1794407755) {
> ...
> Caused by: java.util.concurrent.ExecutionException:
> org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of
> ', )' expected instead of '-'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-4741) AbstractRelNode#getId can overflow into a negative value causing CompileException in certain Enumerable implement methods
[
https://issues.apache.org/jira/browse/CALCITE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L reassigned CALCITE-4741:
--
Assignee: Ruben Q L
> AbstractRelNode#getId can overflow into a negative value causing
> CompileException in certain Enumerable implement methods
> -
>
> Key: CALCITE-4741
> URL: https://issues.apache.org/jira/browse/CALCITE-4741
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Minor
> Fix For: 1.28.0
>
>
> {{AbstractRelNode#getId}} can overflow into a negative value.
> This id is used by the {{implement}} method of several Enumerable operators
> ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to guarantee
> a unique variable name in the dynamic code, e.g.:
> {code:java}
> final ParameterExpression corrArgList =
> Expressions.parameter(Modifier.FINAL,
> List.class, "corrList" + this.getId());
> {code}
> However, if the id is negative, this variable name will have a "-", causing a
> {{CompileException}}:
> {noformat}
> ...
> final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
> org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
> left0, new org.apache.calcite.linq4j.function.Function1() {
> public org.apache.calcite.linq4j.AbstractEnumerable apply(final
> java.util.List corrList-1794407755) {
> ...
> Caused by: java.util.concurrent.ExecutionException:
> org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of
> ', )' expected instead of '-'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4741) AbstractRelNode#getId can overflow into a negative value causing CompileException in certain Enumerable implement methods
[
https://issues.apache.org/jira/browse/CALCITE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-4741.
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/94e2f98d9769ea17b9e8a6a478802f78a745e3f0
> AbstractRelNode#getId can overflow into a negative value causing
> CompileException in certain Enumerable implement methods
> -
>
> Key: CALCITE-4741
> URL: https://issues.apache.org/jira/browse/CALCITE-4741
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{AbstractRelNode#getId}} can overflow into a negative value.
> This id is used by the {{implement}} method of several Enumerable operators
> ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to guarantee
> a unique variable name in the dynamic code, e.g.:
> {code:java}
> final ParameterExpression corrArgList =
> Expressions.parameter(Modifier.FINAL,
> List.class, "corrList" + this.getId());
> {code}
> However, if the id is negative, this variable name will have a "-", causing a
> {{CompileException}}:
> {noformat}
> ...
> final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
> org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
> left0, new org.apache.calcite.linq4j.function.Function1() {
> public org.apache.calcite.linq4j.AbstractEnumerable apply(final
> java.util.List corrList-1794407755) {
> ...
> Caused by: java.util.concurrent.ExecutionException:
> org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of
> ', )' expected instead of '-'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4772) PushProjecto convert bug
[
https://issues.apache.org/jira/browse/CALCITE-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413972#comment-17413972
]
Ruben Q L commented on CALCITE-4772:
[~qianshen] Please set a more specific Jira title, it should describe the bug
(may I suggest something along the lines "PushProject loses RexCall alias"?
Apart from that, I guess the Jira type should be moved from "New feature" to
"Bug".
> PushProjecto convert bug
>
>
> Key: CALCITE-4772
> URL: https://issues.apache.org/jira/browse/CALCITE-4772
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.27.0
>Reporter: qianshen
>Priority: Major
> Attachments: image-2021-09-13-11-36-26-902.png
>
>
>
> {code:java}
> // source sql
> SELECT USER_ID
> ,USER_NAME
> ,DEPT_NO
> ,ROLE_ID
> ,ROLE_NAME
> ,ROLE_NO
> ,ID
> ,NAME
> ,id + age as dt
>
> FROM(
> SELECT user_id,user_name,dept_no
> FROMuser_info
> WHERE dept_no > 20
> ) AS USER_INFO
> LEFT JOIN(
> SELECT role_id,role_name,role_no
> FROMrole_info
> WHERE role_no > 20
> ) AS role_info
> ON USER_ID = ROLE_ID
> LEFT JOIN(
> SELECT id,name,age
> FROMld_ant_test.dim
> WHERE age >22 and (name='a' or name ='b')
> ) dim
> ON dim.id = USER_INFO.USER_ID where user_name='a'
> //execute plan
> LogicalProject(USER_ID=[$0], USER_NAME=[$1], DEPT_NO=[$2], ROLE_ID=[$3],
> ROLE_NAME=[$4], ROLE_NO=[$5], ID=[$6], NAME=[$7], dt=[+($6, $8)])
> LogicalFilter(condition=[=($1, 'a')])
> LogicalJoin(condition=[=($6, $0)], joinType=[left])
> LogicalJoin(condition=[=($0, $3)], joinType=[left])
> LogicalProject(user_id=[$0], user_name=[$1], dept_no=[$2])
> LogicalFilter(condition=[>($2, 20)])
> LogicalTableScan(table=[[LD_ANT_TEST, USER_INFO]])
> LogicalProject(role_id=[$0], role_name=[$1], role_no=[$2])
> LogicalFilter(condition=[>($2, 20)])
> LogicalTableScan(table=[[LD_ANT_TEST, ROLE_INFO]])
> LogicalProject(id=[$0], name=[$1], age=[$2])
> LogicalFilter(condition=[AND(>($2, 22), OR(=($1, 'a'), =($1, 'b')))])
> LogicalTableScan(table=[[LD_ANT_TEST, DIM]])
> {code}
> use HepPlanner RBO rules optimize.
> * CoreRules.FILTER_INTO_JOIN
> * CoreRules.PROJECT_JOIN_TRANSPOSE
>
> After optimize
>
> {code:java}
> //代码占位符
> SELECT *
> FROM (SELECT *
> FROM (SELECT *
> FROM (SELECT `USER_ID` AS `user_id`, `USER_NAME` AS `user_name`, `DEPT_NO` AS
> `dept_no`
> FROM `LD_ANT_TEST`.`USER_INFO`
> WHERE `DEPT_NO` > 20) AS `t0`
> WHERE `user_name` = 'a') AS `t1`
> LEFT JOIN (SELECT `ROLE_ID` AS `role_id`, `ROLE_NAME` AS `role_name`,
> `ROLE_NO` AS `role_no`
> FROM `LD_ANT_TEST`.`ROLE_INFO`
> WHERE `ROLE_NO` > 20) AS `t3` ON `t1`.`user_id` = `t3`.`role_id`) AS `t4`
> LEFT JOIN (SELECT `ID` AS `id`, `NAME` AS `name`, `ID` + `AGE` AS `+`
> FROM `LD_ANT_TEST`.`DIM`
> WHERE `AGE` > 22 AND (`NAME` = 'a' OR `NAME` = 'b')) AS `t7` ON
> `t4`.`user_id` = `t7`.`id`
> LogicalJoin(condition=[=($6, $0)], joinType=[left])
> LogicalProject(user_id=[$0], user_name=[$1], dept_no=[$2], role_id=[$3],
> role_name=[$4], role_no=[$5])
> LogicalJoin(condition=[=($0, $3)], joinType=[left])
> LogicalFilter(condition=[=($1, 'a')])
> LogicalProject(user_id=[$0], user_name=[$1], dept_no=[$2])
> LogicalFilter(condition=[>($2, 20)])
> LogicalTableScan(table=[[LD_ANT_TEST, USER_INFO]])
> LogicalProject(role_id=[$0], role_name=[$1], role_no=[$2])
> LogicalFilter(condition=[>($2, 20)])
> LogicalTableScan(table=[[LD_ANT_TEST, ROLE_INFO]])
> LogicalProject(id=[$0], name=[$1], +=[+($0, $2)])
> LogicalProject(id=[$0], name=[$1], age=[$2])
> LogicalFilter(condition=[AND(>($2, 22), OR(=($1, 'a'), =($1, 'b')))])
> LogicalTableScan(table=[[LD_ANT_TEST, DIM]])
> {code}
>
> In the original SQL, the query column{color:#FF} id + age alias was
> dt{color}. After optimization, {color:#FF}the incorrect alias + was
> used.{color}
> After querying the source code, the problem appears in
> {color:#FF}PushProjector.java#567 {color}
> !image-2021-09-13-11-36-26-902.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule produces result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4773:
---
Description:
The problem can be reproduced with the following test (to be added to
{{unnest.iq}}):
{noformat}
SELECT unnested_outer.val, COUNT(1) AS count_val
FROM
(
SELECT *
FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
AS u(x, y)
) AS t, UNNEST(t.y) AS unnested_outer(val)
WHERE
(
SELECT COUNT(unnested_inner.val) > 0
FROM UNNEST(t.y) AS unnested_inner(val)
WHERE unnested_inner.val = 4
)
GROUP BY unnested_outer.val
ORDER BY count_val DESC, unnested_outer.val ASC;
+-+---+
| VAL | COUNT_VAL |
+-+---+
| 6 | 2 |
| 4 | 1 |
+-+---+
(2 rows)
!ok
{noformat}
When the test is executed, it fails with the following error:
{noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> expression type is RecordType(BOOLEAN $f0) NOT NULL
> set is
> rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
249a254,325
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> at
> org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
...
{noformat}
The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
{{RemoveSingleAggregateRule#onMatch}}:
{code}
// singleAggRel produces a nullable type, so create the new
// projection that casts proj expr to a nullable type.
final RelBuilder relBuilder = call.builder();
final RelDataType type =
relBuilder.getTypeFactory()
.createTypeWithNullability(projExprs.get(0).getType(), true);
final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
projExprs.get(0));
relBuilder.push(aggregate).project(cast);
call.transformTo(relBuilder.build());
{code}
Note that the comment assumes that _"singleAggRel produces a nullable type"_,
but in this particular case, it seems to produce a nullable type, so probably
this piece of code needs to be adapted.
was:
The problem can be reproduced with the following test (to be added to
{{unnest.iq}}):
{noformat}
SELECT unnested_outer.val, COUNT(1) AS count_val
FROM
(
SELECT *
FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
AS u(x, y)
) AS t, UNNEST(t.y) AS unnested_outer(val)
WHERE
(
SELECT COUNT(unnested_inner.val) > 0
FROM UNNEST(t.y) AS unnested_inner(val)
WHERE unnested_inner.val = 4
)
GROUP BY unnested_outer.val
ORDER BY count_val DESC, unnested_outer.val ASC;
+-+---+
| VAL | COUNT_VAL |
+-+---+
| 6 | 2 |
| 4 | 1 |
+-+---+
(2 rows)
!ok
{noformat}
When the test is executed, it fails with the following error:
{noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> expression type is RecordType(BOOLEAN $f0) NOT NULL
> set is
> rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
249a254,325
> at
>
[jira] [Updated] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule produces result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4773:
---
Description:
The problem can be reproduced with the following test (to be added to
{{unnest.iq}}):
{noformat}
SELECT unnested_outer.val, COUNT(1) AS count_val
FROM
(
SELECT *
FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
AS u(x, y)
) AS t, UNNEST(t.y) AS unnested_outer(val)
WHERE
(
SELECT COUNT(unnested_inner.val) > 0
FROM UNNEST(t.y) AS unnested_inner(val)
WHERE unnested_inner.val = 4
)
GROUP BY unnested_outer.val
ORDER BY count_val DESC, unnested_outer.val ASC;
+-+---+
| VAL | COUNT_VAL |
+-+---+
| 6 | 2 |
| 4 | 1 |
+-+---+
(2 rows)
!ok
{noformat}
When the test is executed, it fails with the following error:
{noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> expression type is RecordType(BOOLEAN $f0) NOT NULL
> set is
> rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
249a254,325
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> at
> org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
...
{noformat}
The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
{{RemoveSingleAggregateRule#onMatch}}:
{code:java}
// singleAggRel produces a nullable type, so create the new
// projection that casts proj expr to a nullable type.
final RelBuilder relBuilder = call.builder();
final RelDataType type =
relBuilder.getTypeFactory()
.createTypeWithNullability(projExprs.get(0).getType(), true);
final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
projExprs.get(0));
relBuilder.push(aggregate).project(cast);
call.transformTo(relBuilder.build());
{code}
Note that the comment assumes that _"singleAggRel produces a nullable type"_,
but in this particular case, it seems to produce a non-nullable type, so
probably this piece of code needs to be adapted.
was:
The problem can be reproduced with the following test (to be added to
{{unnest.iq}}):
{noformat}
SELECT unnested_outer.val, COUNT(1) AS count_val
FROM
(
SELECT *
FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
AS u(x, y)
) AS t, UNNEST(t.y) AS unnested_outer(val)
WHERE
(
SELECT COUNT(unnested_inner.val) > 0
FROM UNNEST(t.y) AS unnested_inner(val)
WHERE unnested_inner.val = 4
)
GROUP BY unnested_outer.val
ORDER BY count_val DESC, unnested_outer.val ASC;
+-+---+
| VAL | COUNT_VAL |
+-+---+
| 6 | 2 |
| 4 | 1 |
+-+---+
(2 rows)
!ok
{noformat}
When the test is executed, it fails with the following error:
{noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> expression type is RecordType(BOOLEAN $f0) NOT NULL
> set is
> rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
249a254,325
> at
>
[jira] [Updated] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule produces result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4773:
---
Description:
The problem can be reproduced with the following test (to be added to
{{unnest.iq}}):
{noformat}
SELECT unnested_outer.val, COUNT(1) AS count_val
FROM
(
SELECT *
FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
AS u(x, y)
) AS t, UNNEST(t.y) AS unnested_outer(val)
WHERE
(
SELECT COUNT(unnested_inner.val) > 0
FROM UNNEST(t.y) AS unnested_inner(val)
WHERE unnested_inner.val = 4
)
GROUP BY unnested_outer.val
ORDER BY count_val DESC, unnested_outer.val ASC;
+-+---+
| VAL | COUNT_VAL |
+-+---+
| 6 | 2 |
| 4 | 1 |
+-+---+
(2 rows)
!ok
{noformat}
When the test is executed, it fails with the following error:
{noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> expression type is RecordType(BOOLEAN $f0) NOT NULL
> set is
> rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
249a254,325
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> at
> org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
...
{noformat}
The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
{{RemoveSingleAggregateRule#onMatch}}:
{code}
// singleAggRel produces a nullable type, so create the new
// projection that casts proj expr to a nullable type.
final RelBuilder relBuilder = call.builder();
final RelDataType type =
relBuilder.getTypeFactory()
.createTypeWithNullability(projExprs.get(0).getType(), true);
final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
projExprs.get(0));
relBuilder.push(aggregate).project(cast);
call.transformTo(relBuilder.build());
{code}
Note that the comment assumes that "singleAggRel produces a nullable type", but
in this particular case, it seems to produce a nullable type, so probably this
piece of code needs to be adapted.
was:
The problem can be reproduced with the following test (to be added to
{{unnest.iq}}):
{noformat}
SELECT unnested_outer.val, COUNT(1) AS count_val
FROM
(
SELECT *
FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
AS u(x, y)
) AS t, UNNEST(t.y) AS unnested_outer(val)
WHERE
(
SELECT COUNT(unnested_inner.val) > 0
FROM UNNEST(t.y) AS unnested_inner(val)
WHERE unnested_inner.val = 4
)
GROUP BY unnested_outer.val
ORDER BY count_val DESC, unnested_outer.val ASC;
+-+---+
| VAL | COUNT_VAL |
+-+---+
| 6 | 2 |
| 4 | 1 |
+-+---+
(2 rows)
!ok
{noformat}
When the test is executed, it fails with the following error:
{noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> expression type is RecordType(BOOLEAN $f0) NOT NULL
> set is
> rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
249a254,325
> at
>
[jira] [Updated] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule produces result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4773:
---
Description:
The problem can be reproduced with the following test (to be added to
{{unnest.iq}}):
{noformat}
SELECT unnested_outer.val, COUNT(1) AS count_val
FROM
(
SELECT *
FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
AS u(x, y)
) AS t, UNNEST(t.y) AS unnested_outer(val)
WHERE
(
SELECT COUNT(unnested_inner.val) > 0
FROM UNNEST(t.y) AS unnested_inner(val)
WHERE unnested_inner.val = 4
)
GROUP BY unnested_outer.val
ORDER BY count_val DESC, unnested_outer.val ASC;
+-+---+
| VAL | COUNT_VAL |
+-+---+
| 6 | 2 |
| 4 | 1 |
+-+---+
(2 rows)
!ok
{noformat}
When the test is executed, it fails with the following error:
{noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> expression type is RecordType(BOOLEAN $f0) NOT NULL
> set is
> rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
249a254,325
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> at
> org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
...
{noformat}
The root cause seems to be in this piece of code inside
was:
The problem can be reproduced with the following test (to be added to
{{unnest.iq}}):
{noformat}
SELECT unnested_outer.val, COUNT(1) AS count_val
FROM
(
SELECT *
FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
AS u(x, y)
) AS t, UNNEST(t.y) AS unnested_outer(val)
WHERE
(
SELECT COUNT(unnested_inner.val) > 0
FROM UNNEST(t.y) AS unnested_inner(val)
WHERE unnested_inner.val = 4
)
GROUP BY unnested_outer.val
ORDER BY count_val DESC, unnested_outer.val ASC;
+-+---+
| VAL | COUNT_VAL |
+-+---+
| 6 | 2 |
| 4 | 1 |
+-+---+
(2 rows)
!ok
{noformat}
When the test is executed, it fails with the following error:
{noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> expression type is RecordType(BOOLEAN $f0) NOT NULL
> set is
> rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
249a254,325
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> at
> org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> at
>
[jira] [Created] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule produces result with wrong row type
Ruben Q L created CALCITE-4773:
--
Summary: RelDecorrelator's RemoveSingleAggregateRule produces
result with wrong row type
Key: CALCITE-4773
URL: https://issues.apache.org/jira/browse/CALCITE-4773
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.27.0
Reporter: Ruben Q L
Assignee: Ruben Q L
Fix For: 1.28.0
The problem can be reproduced with the following test (to be added to
{{unnest.iq}}):
{noformat}
SELECT unnested_outer.val, COUNT(1) AS count_val
FROM
(
SELECT *
FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
AS u(x, y)
) AS t, UNNEST(t.y) AS unnested_outer(val)
WHERE
(
SELECT COUNT(unnested_inner.val) > 0
FROM UNNEST(t.y) AS unnested_inner(val)
WHERE unnested_inner.val = 4
)
GROUP BY unnested_outer.val
ORDER BY count_val DESC, unnested_outer.val ASC;
+-+---+
| VAL | COUNT_VAL |
+-+---+
| 6 | 2 |
| 4 | 1 |
+-+---+
(2 rows)
!ok
{noformat}
When the test is executed, it fails with the following error:
{noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> expression type is RecordType(BOOLEAN $f0) NOT NULL
> set is
> rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
249a254,325
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> at
> org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
...
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4773:
---
Summary: RelDecorrelator's RemoveSingleAggregateRule can produce result
with wrong row type (was: RelDecorrelator's RemoveSingleAggregateRule produces
result with wrong row type)
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Fix For: 1.28.0
>
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule produces result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4773:
---
Attachment: CP-22140_fix.patch
> RelDecorrelator's RemoveSingleAggregateRule produces result with wrong row
> type
> ---
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Fix For: 1.28.0
>
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule produces result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4773:
---
Attachment: (was: CP-22140_fix.patch)
> RelDecorrelator's RemoveSingleAggregateRule produces result with wrong row
> type
> ---
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Fix For: 1.28.0
>
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415530#comment-17415530
]
Ruben Q L edited comment on CALCITE-4773 at 9/15/21, 2:23 PM:
--
[~julianhyde], sorry to insist, but it seems to me that the logical plan (and
rowTypes) in this case are correct.
We are dealing with this plan:
{noformat}
LogicalAggregate(group=[{}], agg#0=[SINGLE_VALUE($0)])
LogicalProject(EXPR$0=[>($0, 0)])
LogicalAggregate(group=[{}], agg#0=[COUNT()])
LogicalFilter(condition=[=($0, 4)])
LogicalProject(VAL=[$0])
Uncollect
LogicalProject(Y=[$cor1.Y])
LogicalValues(tuples=[[{ 0 }]])
RowTypes:
LogicalAggregate(SINGLE_VALUE): RecordType(BOOLEAN NOT NULL $f0) NOT NULL
LogicalProject: RecordType(BOOLEAN NOT NULL EXPR$0) NOT NULL
LogicalAggregate(COUNT): RecordType(BIGINT NOT NULL $f0) NOT NULL
...
{noformat}
The return type of the count is bigint *non-nullable* (see
{{SqlCountAggFunction}}); which makes sense, since a COUNT can never return
null (it can return zero, but not null), is that correct? Thus, the return type
of the projection ($0 > 0) will consequently be a boolean non-null; and finally
the single_value function has a return type inference based on its argument
(see {{SqlSingleValueAggFunction}}), so we get the top boolean non-null.
Therefore, it would seem that the current precondition in {{RelDecorrelator}}'s
{{RemoveSingleAggregateRule#onMatch}}: _"singleAggRel produces a nullable
type..."_ has a loophole, and it can (in certain cases, at least this one) to
produce a non-nullable type, so the proposed PR could be a valid solution for
the issue.
was (Author: rubenql):
[~julianhyde], sorry to insist, but it seems to me that the logical plan (and
rowTypes) in this case are correct.
We are dealing with an Aggregate that contains as agg function a COUNT:
{noformat}
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
{noformat}
and the return type of the count is BIGINT *non-nullable* (see
{{SqlCountAggFunction}}); which makes sense, since a COUNT can never return
null (it can return zero, but not null), is that correct?
Therefore, it would seem that the current precondition in
{{RelDecorrelator}}'s {{RemoveSingleAggregateRule#onMatch}}: _"singleAggRel
produces a nullable type..."_ has a loophole, and it can (in certain cases, at
least this one) to produce a non-nullable type, so the proposed PR could be a
valid solution for the issue.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> >
[jira] [Commented] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415530#comment-17415530
]
Ruben Q L commented on CALCITE-4773:
[~julianhyde], sorry to insist, but it seems to me that the logical plan (and
rowTypes) in this case are correct.
We are dealing with an Aggregate that contains as agg function a COUNT:
{noformat}
> expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> LogicalAggregate(group=[{}], agg#0=[COUNT()])
> LogicalFilter(condition=[=($0, 4)])
> LogicalProject(VAL=[$0])
> Uncollect
> LogicalProject(Y=[$cor1.Y])
> LogicalValues(tuples=[[{ 0 }]])
{noformat}
and the return type of the count is BIGINT *non-nullable* (see
{{SqlCountAggFunction}}); which makes sense, since a COUNT can never return
null (it can return zero, but not null), is that correct?
Therefore, it would seem that the current precondition in
{{RelDecorrelator}}'s {{RemoveSingleAggregateRule#onMatch}}: _"singleAggRel
produces a nullable type..."_ has a loophole, and it can (in certain cases, at
least this one) to produce a non-nullable type, so the proposed PR could be a
valid solution for the issue.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
>
[jira] [Comment Edited] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414767#comment-17414767
]
Ruben Q L edited comment on CALCITE-4773 at 9/14/21, 7:16 AM:
--
{quote}No, it always produces a nullable type ...
{quote}
But in this case it does not. So, what's wrong here?
Just to clarify, you think that the sql query (which of course is a simplified
version of an actual query that fetches items from a table instead of a
{{values}}) is wrong (even though it passes Calcite's sql parser and
SqlToRelConverter)? Or that the sql query is right, but somehow there is a
problem in SqlToRelConverter when generating the row type of the aggregation?
I have just tested the exact same query in PostgreSQL, and it returns
successfully the expected result, so the query itself seems to be a valid one.
BTW, I already found a workaround: re-writing the query as {{... WHERE EXISTS
(SELECT 1 ...}} instead of {{...WHERE (SELECT COUNT ...}} avoids the issue, but
if the original query with {{COUNT}} is a valid one, I guess the issue would
need to be solved somehow.
was (Author: rubenql):
{quote}No, it always produces a nullable type ...
{quote}
But in this case it does not. So, what's wrong here?
Just to clarify, you think that the sql query (which of course is a simplified
version of an actual query that fetches items from a table instead of a
{{values}}) is wrong (even though it passes Calcite's sql parser and
SqlToRelConverter)? Or that the sql query is right, but somehow there is a
problem in SqlToRelConverter when generating the row type of the aggregation?
I have just tested the exact same query in PostgreSQL, and it returns
successfully the expected result, so the query itself seems to be a valid one.
BTW, I already found a workaround: re-writing the query as {{... WHERE EXISTS
(SELECT 1 ...}} instead of {{...WHERE (SELECT COUNT ... }} avoids the issue,
but if the original query with {{COUNT}} is a valid one, I guess the issue
would need to be solved somehow.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> >
[jira] [Commented] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414767#comment-17414767
]
Ruben Q L commented on CALCITE-4773:
{quote}No, it always produces a nullable type ...
{quote}
But in this case it does not. So, what's wrong here?
Just to clarify, you think that the sql query (which of course is a simplified
version of an actual query that fetches items from a table instead of a
{{values}}) is wrong (even though it passes Calcite's sql parser and
SqlToRelConverter)? Or that the sql query is right, but somehow there is a
problem in SqlToRelConverter when generating the row type of the aggregation?
I have just tested the exact same query in PostgreSQL, and it returns
successfully the expected result, so the query itself seems to be a valid one.
BTW, I already found a workaround: re-writing the query as {{... WHERE EXISTS
(SELECT 1 ...}} instead of {{...WHERE (SELECT COUNT ... }} avoids the issue,
but if the original query with {{COUNT}} is a valid one, I guess the issue
would need to be solved somehow.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast =
[jira] [Commented] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415623#comment-17415623
]
Ruben Q L commented on CALCITE-4773:
Yes, I manages to reproduce the issue with a simpler test:
{noformat}
# [CALCITE-4773] RelDecorrelator's RemoveSingleAggregateRule produces result
with wrong row type
!use bookstore
SELECT au."name"
FROM "bookstore"."authors" au
WHERE (
SELECT COUNT(*) > 0
FROM UNNEST(au."books") AS "unnested"("title", "year", "pages")
WHERE "unnested"."year" < 1920
);
+-+
| name|
+-+
| Victor Hugo |
+-+
(1 row)
!ok
{noformat}
What do you think [~julianhyde]?
I'll update the PR as soon as I can.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This
[jira] [Comment Edited] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415623#comment-17415623
]
Ruben Q L edited comment on CALCITE-4773 at 9/15/21, 4:16 PM:
--
Yes, I managed to reproduce the issue with a simpler test:
{noformat}
# [CALCITE-4773] RelDecorrelator's RemoveSingleAggregateRule produces result
with wrong row type
!use bookstore
SELECT au."name"
FROM "bookstore"."authors" au
WHERE (
SELECT COUNT(*) > 0
FROM UNNEST(au."books") AS "unnested"("title", "year", "pages")
WHERE "unnested"."year" < 1920
);
+-+
| name|
+-+
| Victor Hugo |
+-+
(1 row)
!ok
{noformat}
What do you think [~julianhyde]?
I'll update the PR as soon as I can.
was (Author: rubenql):
Yes, I manages to reproduce the issue with a simpler test:
{noformat}
# [CALCITE-4773] RelDecorrelator's RemoveSingleAggregateRule produces result
with wrong row type
!use bookstore
SELECT au."name"
FROM "bookstore"."authors" au
WHERE (
SELECT COUNT(*) > 0
FROM UNNEST(au."books") AS "unnested"("title", "year", "pages")
WHERE "unnested"."year" < 1920
);
+-+
| name|
+-+
| Victor Hugo |
+-+
(1 row)
!ok
{noformat}
What do you think [~julianhyde]?
I'll update the PR as soon as I can.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection
[jira] [Commented] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415933#comment-17415933
]
Ruben Q L commented on CALCITE-4773:
Unfortunately, that type of query does not seem to generate a logical plan with
the pattern SINGLE_AGGREGATE + PROJECT + AGGREGATE that fires
{{RelDecorrelator}}'s {{RemoveSingleAggregateRule}}, so it does not reproduce
the issue,
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4777) Casting from decimal to boolean throws an exception
[
https://issues.apache.org/jira/browse/CALCITE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416105#comment-17416105
]
Ruben Q L commented on CALCITE-4777:
Thanks for the investigation [~xuyangzhong]. So it seems we are dealing with
two issues here:
- casting from decimal to boolean throws an exception
- casting from any other numeric type to boolean returns always false
> Casting from decimal to boolean throws an exception
> ---
>
> Key: CALCITE-4777
> URL: https://issues.apache.org/jira/browse/CALCITE-4777
> Project: Calcite
> Issue Type: Bug
>Reporter: xuyangzhong
>Priority: Major
> Attachments: calcite.png
>
>
> My sql is the following:
> {code:java}
> // code placeholder
> select * from test where cast (0.10915913549909961 as boolean){code}
>
> I want to simplify the cast. An exception is thrown:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.lang.RuntimeException: while resolving method
> 'booleanValue' in class class java.math.BigDecimal at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:424) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:435) at
> org.apache.calcite.linq4j.tree.Expressions.unbox(Expressions.java:1453) at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:398)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:326)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateCast(RexToLixTranslator.java:543)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$CastImplementor.implementSafe(RexImpTable.java:2450)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.genValueStatement(RexImpTable.java:2894)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.implement(RexImpTable.java:2859)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:1089)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:174) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:975)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexLocalRef.accept(RexLocalRef.java:75) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:237)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:231)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateList(RexToLixTranslator.java:823)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateProjects(RexToLixTranslator.java:198)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:90)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:66) at
> org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:128) at
> org.apache.calcite.rex.RexSimplify.simplifyCast(RexSimplify.java:2101) at
> org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:326) at
> org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:287) at
> org.apache.flink.table.examples.java.tests.CalciteTest.main(CalciteTest.java:47)
> Caused by: java.lang.NoSuchMethodException:
> java.math.BigDecimal.booleanValue() at
> java.lang.Class.getMethod(Class.java:1786) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:421) ... 25
> more
> {code}
> In order to avoid that I used the wrong rule or it caused by my bad coding, i
> write the test case following:
>
> {code:java}
> // code placeholder
> JavaTypeFactory typeFactory = new
> JavaTypeFactoryImpl(RelDataTypeSystem.DEFAULT);
> RexBuilder rexBuilder = new RexBuilder(typeFactory);
> final RexSimplify simplify = new RexSimplify(rexBuilder,
> RelOptPredicateList.EMPTY, RexUtil.EXECUTOR);
> RelDataType type = new BasicSqlType(typeFactory.getTypeSystem(),
> SqlTypeName.BOOLEAN);
> RelDataType bb = new
> BasicSqlType(typeFactory.getTypeSystem(),SqlTypeName.DECIMAL,18,17);
> SqlOperator op = new SqlCastFunction();
> RexLiteral lt =
> rexBuilder.makeExactLiteral(BigDecimal.valueOf(0.10915913549909961),bb);
> List list = new ArrayList<>();
> list.add(lt);
> RexNode rexNode = rexBuilder.makeCall(type,op,list);
> simplify.simplifyUnknownAs(rexNode, RexUnknownAs.UNKNOWN);
> {code}
> and it throws the same exception.
>
> Actually, the cast simplify operation will enter the function _translateCast_
> in _RexToLixTranslator_.It misses the "case BOOLEAN" and uses the convert in
> EnumUtils. However, because the Decimal's Primitive is
[jira] [Commented] (CALCITE-4777) Casting from decimal to boolean throws an exception
[
https://issues.apache.org/jira/browse/CALCITE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416184#comment-17416184
]
Ruben Q L commented on CALCITE-4777:
I guess it's a design decision, we should consider:
- From which types do we want to support a cast into boolean
- Ensure the operation works fine (I assume: zero -> false; different from
zero -> true ?)
- Ensure Calcite doc / website is updated accordingly (it seems the current
matrix in the site is not correct)
> Casting from decimal to boolean throws an exception
> ---
>
> Key: CALCITE-4777
> URL: https://issues.apache.org/jira/browse/CALCITE-4777
> Project: Calcite
> Issue Type: Bug
>Reporter: xuyangzhong
>Priority: Major
> Attachments: calcite.png
>
>
> My sql is the following:
> {code:java}
> // code placeholder
> select * from test where cast (0.10915913549909961 as boolean){code}
>
> I want to simplify the cast. An exception is thrown:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.lang.RuntimeException: while resolving method
> 'booleanValue' in class class java.math.BigDecimal at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:424) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:435) at
> org.apache.calcite.linq4j.tree.Expressions.unbox(Expressions.java:1453) at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:398)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:326)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateCast(RexToLixTranslator.java:543)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$CastImplementor.implementSafe(RexImpTable.java:2450)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.genValueStatement(RexImpTable.java:2894)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.implement(RexImpTable.java:2859)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:1089)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:174) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:975)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexLocalRef.accept(RexLocalRef.java:75) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:237)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:231)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateList(RexToLixTranslator.java:823)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateProjects(RexToLixTranslator.java:198)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:90)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:66) at
> org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:128) at
> org.apache.calcite.rex.RexSimplify.simplifyCast(RexSimplify.java:2101) at
> org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:326) at
> org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:287) at
> org.apache.flink.table.examples.java.tests.CalciteTest.main(CalciteTest.java:47)
> Caused by: java.lang.NoSuchMethodException:
> java.math.BigDecimal.booleanValue() at
> java.lang.Class.getMethod(Class.java:1786) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:421) ... 25
> more
> {code}
> In order to avoid that I used the wrong rule or it caused by my bad coding, i
> write the test case following:
>
> {code:java}
> // code placeholder
> JavaTypeFactory typeFactory = new
> JavaTypeFactoryImpl(RelDataTypeSystem.DEFAULT);
> RexBuilder rexBuilder = new RexBuilder(typeFactory);
> final RexSimplify simplify = new RexSimplify(rexBuilder,
> RelOptPredicateList.EMPTY, RexUtil.EXECUTOR);
> RelDataType type = new BasicSqlType(typeFactory.getTypeSystem(),
> SqlTypeName.BOOLEAN);
> RelDataType bb = new
> BasicSqlType(typeFactory.getTypeSystem(),SqlTypeName.DECIMAL,18,17);
> SqlOperator op = new SqlCastFunction();
> RexLiteral lt =
> rexBuilder.makeExactLiteral(BigDecimal.valueOf(0.10915913549909961),bb);
> List list = new ArrayList<>();
> list.add(lt);
> RexNode rexNode = rexBuilder.makeCall(type,op,list);
> simplify.simplifyUnknownAs(rexNode, RexUnknownAs.UNKNOWN);
> {code}
> and it throws the same exception.
>
> Actually, the cast simplify operation will enter the function _translateCast_
> in _RexToLixTranslator_.It misses
[jira] [Commented] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17417499#comment-17417499
]
Ruben Q L commented on CALCITE-4773:
[~julianhyde], do you have further comments about this ticket? Otherwise I
propose to squash commits and merge the PR in the coming days.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416539#comment-17416539
]
Ruben Q L commented on CALCITE-4773:
Sure, I have committed a refactoring + some comment improvements.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4777) Casting from decimal to boolean throws an exception
[
https://issues.apache.org/jira/browse/CALCITE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416035#comment-17416035
]
Ruben Q L commented on CALCITE-4777:
Interestingly, it seems that using scientific notation e.g. {{CAST (0.42e0 AS
BOOLEAN)}} instead of {{CAST (0.42 AS BOOLEAN)}} avoids the exception (but it
seems to return always {{false}}, no matter the number, as explained by
[~nobigo]). Do you confirm [~xuyangzhong]?
> Casting from decimal to boolean throws an exception
> ---
>
> Key: CALCITE-4777
> URL: https://issues.apache.org/jira/browse/CALCITE-4777
> Project: Calcite
> Issue Type: Bug
>Reporter: xuyangzhong
>Priority: Major
> Attachments: calcite.png
>
>
> My sql is the following:
> {code:java}
> // code placeholder
> select * from test where cast (0.10915913549909961 as boolean){code}
>
> I want to simplify the cast. An exception is thrown:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.lang.RuntimeException: while resolving method
> 'booleanValue' in class class java.math.BigDecimal at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:424) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:435) at
> org.apache.calcite.linq4j.tree.Expressions.unbox(Expressions.java:1453) at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:398)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:326)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateCast(RexToLixTranslator.java:543)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$CastImplementor.implementSafe(RexImpTable.java:2450)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.genValueStatement(RexImpTable.java:2894)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.implement(RexImpTable.java:2859)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:1089)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:174) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:975)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexLocalRef.accept(RexLocalRef.java:75) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:237)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:231)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateList(RexToLixTranslator.java:823)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateProjects(RexToLixTranslator.java:198)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:90)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:66) at
> org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:128) at
> org.apache.calcite.rex.RexSimplify.simplifyCast(RexSimplify.java:2101) at
> org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:326) at
> org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:287) at
> org.apache.flink.table.examples.java.tests.CalciteTest.main(CalciteTest.java:47)
> Caused by: java.lang.NoSuchMethodException:
> java.math.BigDecimal.booleanValue() at
> java.lang.Class.getMethod(Class.java:1786) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:421) ... 25
> more
> {code}
> In order to avoid that I used the wrong rule or it caused by my bad coding, i
> write the test case following:
>
> {code:java}
> // code placeholder
> JavaTypeFactory typeFactory = new
> JavaTypeFactoryImpl(RelDataTypeSystem.DEFAULT);
> RexBuilder rexBuilder = new RexBuilder(typeFactory);
> final RexSimplify simplify = new RexSimplify(rexBuilder,
> RelOptPredicateList.EMPTY, RexUtil.EXECUTOR);
> RelDataType type = new BasicSqlType(typeFactory.getTypeSystem(),
> SqlTypeName.BOOLEAN);
> RelDataType bb = new
> BasicSqlType(typeFactory.getTypeSystem(),SqlTypeName.DECIMAL,18,17);
> SqlOperator op = new SqlCastFunction();
> RexLiteral lt =
> rexBuilder.makeExactLiteral(BigDecimal.valueOf(0.10915913549909961),bb);
> List list = new ArrayList<>();
> list.add(lt);
> RexNode rexNode = rexBuilder.makeCall(type,op,list);
> simplify.simplifyUnknownAs(rexNode, RexUnknownAs.UNKNOWN);
> {code}
> and it throws the same exception.
>
> Actually, the cast simplify operation will enter the function _translateCast_
> in _RexToLixTranslator_.It misses the "case BOOLEAN" and uses the convert in
>
[jira] [Commented] (CALCITE-4777) Casting from decimal to boolean throws an exception
[
https://issues.apache.org/jira/browse/CALCITE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415431#comment-17415431
]
Ruben Q L commented on CALCITE-4777:
Is casting decimal to boolean something standard? I just tried a similar query
in PostgreSQL and it did not work either (it failed more gracefully with a
message "ERROR: cannot cast type numeric to boolean"). Perhaps the proper way
to deal with this situation is using a {{CASE WHEN ...}} ?
> Casting from decimal to boolean throws an exception
> ---
>
> Key: CALCITE-4777
> URL: https://issues.apache.org/jira/browse/CALCITE-4777
> Project: Calcite
> Issue Type: Bug
>Reporter: xuyangzhong
>Priority: Major
>
> My sql is the following:
> {code:java}
> // code placeholder
> select * from test where cast (0.10915913549909961 as boolean){code}
>
> I want to simplify the cast. An exception is thrown:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.lang.RuntimeException: while resolving method
> 'booleanValue' in class class java.math.BigDecimal at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:424) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:435) at
> org.apache.calcite.linq4j.tree.Expressions.unbox(Expressions.java:1453) at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:398)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:326)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateCast(RexToLixTranslator.java:543)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$CastImplementor.implementSafe(RexImpTable.java:2450)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.genValueStatement(RexImpTable.java:2894)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.implement(RexImpTable.java:2859)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:1089)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:174) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:975)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexLocalRef.accept(RexLocalRef.java:75) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:237)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:231)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateList(RexToLixTranslator.java:823)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateProjects(RexToLixTranslator.java:198)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:90)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:66) at
> org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:128) at
> org.apache.calcite.rex.RexSimplify.simplifyCast(RexSimplify.java:2101) at
> org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:326) at
> org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:287) at
> org.apache.flink.table.examples.java.tests.CalciteTest.main(CalciteTest.java:47)
> Caused by: java.lang.NoSuchMethodException:
> java.math.BigDecimal.booleanValue() at
> java.lang.Class.getMethod(Class.java:1786) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:421) ... 25
> more
> {code}
> In order to avoid that I used the wrong rule or it caused by my bad coding, i
> write the test case following:
>
> {code:java}
> // code placeholder
> JavaTypeFactory typeFactory = new
> JavaTypeFactoryImpl(RelDataTypeSystem.DEFAULT);
> RexBuilder rexBuilder = new RexBuilder(typeFactory);
> final RexSimplify simplify = new RexSimplify(rexBuilder,
> RelOptPredicateList.EMPTY, RexUtil.EXECUTOR);
> RelDataType type = new BasicSqlType(typeFactory.getTypeSystem(),
> SqlTypeName.BOOLEAN);
> RelDataType bb = new
> BasicSqlType(typeFactory.getTypeSystem(),SqlTypeName.DECIMAL,18,17);
> SqlOperator op = new SqlCastFunction();
> RexLiteral lt =
> rexBuilder.makeExactLiteral(BigDecimal.valueOf(0.10915913549909961),bb);
> List list = new ArrayList<>();
> list.add(lt);
> RexNode rexNode = rexBuilder.makeCall(type,op,list);
> simplify.simplifyUnknownAs(rexNode, RexUnknownAs.UNKNOWN);
> {code}
> and it throws the same exception.
>
> Actually, the cast simplify operation will enter the function _translateCast_
> in _RexToLixTranslator_.It misses the "case BOOLEAN" and uses the convert in
> EnumUtils.
[jira] [Updated] (CALCITE-4848) Adding a HAVING condition to a query with a dynamic parameter makes the result empty
[
https://issues.apache.org/jira/browse/CALCITE-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4848:
---
Fix Version/s: 1.28.0
> Adding a HAVING condition to a query with a dynamic parameter makes the
> result empty
>
>
> Key: CALCITE-4848
> URL: https://issues.apache.org/jira/browse/CALCITE-4848
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.27.0
>Reporter: Thomas Rebele
>Priority: Major
> Fix For: 1.28.0
>
> Attachments: CALCITE-4848.patch
>
>
> The query
> {code:java}
> SELECT sal, COUNT(1) AS count_val
> FROM emp t WHERE sal = ?
> GROUP BY sal HAVING sal < 1000 {code}
> gets translated to the following logical plan:
> {code:java}
> LogicalFilter(condition=[<($0, 1000)])
> LogicalAggregate(group=[{0}], COUNT_VAL=[COUNT()])
> LogicalProject(SAL=[$5], $f1=[1])
> LogicalFilter(condition=[=($5, ?0)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]]) {code}
> However, applying CoreRules.FILTER_REDUCE_EXPRESSIONS to the plan it just
> becomes
> {code:java}
> LogicalValues(tuples=[[]]) {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4848) Adding a HAVING condition to a query with a dynamic parameter makes the result always empty
[
https://issues.apache.org/jira/browse/CALCITE-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-4848.
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/23e1b1fcae6614fb9b6600951251ba622b379b30
Thanks [~thomas.rebele] for the PR!
> Adding a HAVING condition to a query with a dynamic parameter makes the
> result always empty
> ---
>
> Key: CALCITE-4848
> URL: https://issues.apache.org/jira/browse/CALCITE-4848
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.27.0
>Reporter: Thomas Rebele
>Assignee: Thomas Rebele
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Attachments: CALCITE-4848.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The query
> {code:java}
> SELECT sal, COUNT(1) AS count_val
> FROM emp t WHERE sal = ?
> GROUP BY sal HAVING sal < 1000 {code}
> gets translated to the following logical plan:
> {code:java}
> LogicalFilter(condition=[<($0, 1000)])
> LogicalAggregate(group=[{0}], COUNT_VAL=[COUNT()])
> LogicalProject(SAL=[$5], $f1=[1])
> LogicalFilter(condition=[=($5, ?0)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]]) {code}
> However, applying CoreRules.FILTER_REDUCE_EXPRESSIONS to the plan it just
> becomes
> {code:java}
> LogicalValues(tuples=[[]]) {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4809) Mixed merge join result is incomplete
[ https://issues.apache.org/jira/browse/CALCITE-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421268#comment-17421268 ] Ruben Q L commented on CALCITE-4809: Can you provide a unit test? [MergeJoin implementation has a prerrequisite|https://github.com/apache/calcite/blob/219e41eab20533f02d41238d1ebc617cc813b9a2/linq4j/src/main/java/org/apache/calcite/linq4j/EnumerableDefaults.java#L2208]: inputs sorted in ascending order with nulls last. I have the impression that in your example this is not respected: after the left t1xt2 there should have been a sort by t2.id nulls last, which will lead to 4,4 being processed before 3,null; which would lead to the correct result. > Mixed merge join result is incomplete > -- > > Key: CALCITE-4809 > URL: https://issues.apache.org/jira/browse/CALCITE-4809 > Project: Calcite > Issue Type: Bug > Components: linq4j >Affects Versions: 1.26.0 >Reporter: super486 >Priority: Major > > The result of mixed merge join is incomplete. The merge join relies on the > row data of the two tables to be sorted by the join field, but if the left > table itself is a left join, nulls may appear due to mismatches, causing the > entire join to end early。 > EG： > t1 left join t2 on t1.id=t2.id join t3 on t2.id=t3.id; > t1 rows: 1,2,3,4 > t2 rows: 1,2,4 > t3 rows: 1.2.3.4 > final result is:1,2. but 4 is lost; > when MergeJoinEnumerator left (t1xt2) come to: 3,null, right (t3) is 3, > the null <3 cause left advance finish wrong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4809) Mixed merge join result is incomplete
[ https://issues.apache.org/jira/browse/CALCITE-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421268#comment-17421268 ] Ruben Q L edited comment on CALCITE-4809 at 9/28/21, 9:11 AM: -- Can you provide a unit test? [MergeJoin implementation has a prerrequisite|https://github.com/apache/calcite/blob/219e41eab20533f02d41238d1ebc617cc813b9a2/linq4j/src/main/java/org/apache/calcite/linq4j/EnumerableDefaults.java#L2208]: inputs sorted (by join key) in ascending order with nulls last. I have the impression that in your example this is not respected: after the left t1xt2 there should have been a sort by t2.id nulls last, which will lead to 4,4 being processed before 3,null; which would lead to the correct result. was (Author: rubenql): Can you provide a unit test? [MergeJoin implementation has a prerrequisite|https://github.com/apache/calcite/blob/219e41eab20533f02d41238d1ebc617cc813b9a2/linq4j/src/main/java/org/apache/calcite/linq4j/EnumerableDefaults.java#L2208]: inputs sorted in ascending order with nulls last. I have the impression that in your example this is not respected: after the left t1xt2 there should have been a sort by t2.id nulls last, which will lead to 4,4 being processed before 3,null; which would lead to the correct result. > Mixed merge join result is incomplete > -- > > Key: CALCITE-4809 > URL: https://issues.apache.org/jira/browse/CALCITE-4809 > Project: Calcite > Issue Type: Bug > Components: linq4j >Affects Versions: 1.26.0 >Reporter: super486 >Priority: Major > > The result of mixed merge join is incomplete. The merge join relies on the > row data of the two tables to be sorted by the join field, but if the left > table itself is a left join, nulls may appear due to mismatches, causing the > entire join to end early。 > EG： > t1 left join t2 on t1.id=t2.id join t3 on t2.id=t3.id; > t1 rows: 1,2,3,4 > t2 rows: 1,2,4 > t3 rows: 1.2.3.4 > final result is:1,2. but 4 is lost; > when MergeJoinEnumerator left (t1xt2) come to: 3,null, right (t3) is 3, > the null <3 cause left advance finish wrong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4777) Casting from DECIMAL to BOOLEAN throws an exception
[
https://issues.apache.org/jira/browse/CALCITE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422563#comment-17422563
]
Ruben Q L commented on CALCITE-4777:
[~julianhyde], if I understood this correctly, the purpose of the PR will be:
- Fail gracefully with an adequate error message when trying to cast a decimal
(or float, real, double) into boolean
- Update the cast-compatibility matrix in Calcite site to reflect that these
castings are not allowed
> Casting from DECIMAL to BOOLEAN throws an exception
> ---
>
> Key: CALCITE-4777
> URL: https://issues.apache.org/jira/browse/CALCITE-4777
> Project: Calcite
> Issue Type: Bug
>Reporter: xuyangzhong
>Assignee: xuyangzhong
>Priority: Major
> Labels: pull-request-available
> Attachments: calcite.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> My sql is the following:
> {code:java}
> // code placeholder
> select * from test where cast (0.10915913549909961 as boolean){code}
>
> I want to simplify the cast. An exception is thrown:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.lang.RuntimeException: while resolving method
> 'booleanValue' in class class java.math.BigDecimal at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:424) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:435) at
> org.apache.calcite.linq4j.tree.Expressions.unbox(Expressions.java:1453) at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:398)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:326)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateCast(RexToLixTranslator.java:543)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$CastImplementor.implementSafe(RexImpTable.java:2450)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.genValueStatement(RexImpTable.java:2894)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.implement(RexImpTable.java:2859)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:1089)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:174) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:975)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexLocalRef.accept(RexLocalRef.java:75) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:237)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:231)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateList(RexToLixTranslator.java:823)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateProjects(RexToLixTranslator.java:198)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:90)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:66) at
> org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:128) at
> org.apache.calcite.rex.RexSimplify.simplifyCast(RexSimplify.java:2101) at
> org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:326) at
> org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:287) at
> org.apache.flink.table.examples.java.tests.CalciteTest.main(CalciteTest.java:47)
> Caused by: java.lang.NoSuchMethodException:
> java.math.BigDecimal.booleanValue() at
> java.lang.Class.getMethod(Class.java:1786) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:421) ... 25
> more
> {code}
> In order to avoid that I used the wrong rule or it caused by my bad coding, i
> write the test case following:
>
> {code:java}
> // code placeholder
> JavaTypeFactory typeFactory = new
> JavaTypeFactoryImpl(RelDataTypeSystem.DEFAULT);
> RexBuilder rexBuilder = new RexBuilder(typeFactory);
> final RexSimplify simplify = new RexSimplify(rexBuilder,
> RelOptPredicateList.EMPTY, RexUtil.EXECUTOR);
> RelDataType type = new BasicSqlType(typeFactory.getTypeSystem(),
> SqlTypeName.BOOLEAN);
> RelDataType bb = new
> BasicSqlType(typeFactory.getTypeSystem(),SqlTypeName.DECIMAL,18,17);
> SqlOperator op = new SqlCastFunction();
> RexLiteral lt =
> rexBuilder.makeExactLiteral(BigDecimal.valueOf(0.10915913549909961),bb);
> List list = new ArrayList<>();
> list.add(lt);
> RexNode rexNode = rexBuilder.makeCall(type,op,list);
> simplify.simplifyUnknownAs(rexNode, RexUnknownAs.UNKNOWN);
> {code}
> and it throws the same exception.
>
>
[jira] [Commented] (CALCITE-4810) Failed to cast a double field (which isn't a literal) to boolean in the test case.
[
https://issues.apache.org/jira/browse/CALCITE-4810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422607#comment-17422607
]
Ruben Q L commented on CALCITE-4810:
There are discussions in CALCITE-4777 to decide whether or not we may want to
forbid casting DOUBLE (and FLOAT/REAL) into BOOLEAN.
> Failed to cast a double field (which isn't a literal) to boolean in the test
> case.
> --
>
> Key: CALCITE-4810
> URL: https://issues.apache.org/jira/browse/CALCITE-4810
> Project: Calcite
> Issue Type: Bug
>Reporter: xuyangzhong
>Priority: Minor
>
> The cause of this matter is that when I add a test case in
> SqlOperatorBaseTest:
> {code:java}
> // code placeholder
> @Test void myTest(){
> tester.checkBoolean("cast(0.1e0 as boolean)",false);
> }
> {code}
> The tester will turn the single sql to these two SQLs which will be all
> tested:
> {code:java}
> // SQL 1
> values (cast(0.1e0 as boolean))
> // SQL 2
> select cast(p0 as boolean) from (values (0.1e0)) as t(p0)
> {code}
> The SQL1 passed by the RexSimplify. But The SQL2 throws a exception following:
> {code:java}
> // code placeholder
> java.sql.SQLException: Error while executing SQL "select cast(p0 as boolean)
> from (values (0.1e0)) as t(p0)": Error while compiling generated Java code:
> public org.apache.calcite.linq4j.Enumerable bind(final
> org.apache.calcite.DataContext root) {
> final org.apache.calcite.linq4j.Enumerable _inputEnumerable =
> org.apache.calcite.linq4j.Linq4j.asEnumerable(new Double[] {
> 0.1D});
> return new org.apache.calcite.linq4j.AbstractEnumerable(){
> public org.apache.calcite.linq4j.Enumerator enumerator() {
> return new org.apache.calcite.linq4j.Enumerator(){
> public final org.apache.calcite.linq4j.Enumerator
> inputEnumerator = _inputEnumerable.enumerator();
> public void reset() {
> inputEnumerator.reset();
> }
>
> public boolean moveNext() {
> return inputEnumerator.moveNext();
> }
>
> public void close() {
> inputEnumerator.close();
> }
>
> public Object current() {
> return (boolean)
> org.apache.calcite.runtime.SqlFunctions.toDouble(inputEnumerator.current());
> }
>
> };
> }
>
> };
> }
>
>
> public Class getElementType() {
> return boolean.class;
> }
>
>
>
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
> at
> org.apache.calcite.sql.test.SqlOperatorBaseTest$TesterImpl.check(SqlOperatorBaseTest.java:10371)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.check(AbstractSqlTester.java:464)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.checkBoolean(AbstractSqlTester.java:433)
> at
> org.apache.calcite.sql.test.SqlOperatorBaseTest.myTest(SqlOperatorBaseTest.java:1619)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
> at
> org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
> at
>
[jira] [Commented] (CALCITE-4834) JaninoRelMetadataProvider uses hardcoded class name
[
https://issues.apache.org/jira/browse/CALCITE-4834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425047#comment-17425047
]
Ruben Q L commented on CALCITE-4834:
FYI [~jamesstarr], [~jcamachorodriguez]
> JaninoRelMetadataProvider uses hardcoded class name
> ---
>
> Key: CALCITE-4834
> URL: https://issues.apache.org/jira/browse/CALCITE-4834
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Fix For: 1.28.0
>
>
> This seems caused by CALCITE-4546
> ([fbbdf46|https://github.com/apache/calcite/commit/fbbdf465df46b4e6f9863d7d1dfdcb19a43f2032])
> {{JaninoRelMetadataProvider}} lines 135 & 141 is using (for the generated
> code) the hardcoded name of the class {{MetadataDef}}:
> {code}
> buff.append(" private final org.apache.calcite.rel.metadata.MetadataDef
> def;\n");
> for (Map.Entry, String> handlerAndName :
> handlerToName.entrySet()) {
> ...
> .append(" org.apache.calcite.rel.metadata.MetadataDef def");
> {code}
> This can lead to issues (e.g. if a downstream project shades Calcite
> library). The safer way to do this is using {{Class#getName}}, as it is
> already done in the rest of the code, e.g. in
> {{JaninoRelMetadataProvider:158}}:
> {code}
> .append(MetadataDef.class.getName())
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4834) JaninoRelMetadataProvider uses hardcoded class name
[
https://issues.apache.org/jira/browse/CALCITE-4834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4834:
---
Description:
This seems caused by CALCITE-4546
([fbbdf46|https://github.com/apache/calcite/commit/fbbdf465df46b4e6f9863d7d1dfdcb19a43f2032])
{{JaninoRelMetadataProvider}} lines 135 & 141 is using (for the generated code)
the hardcoded name of the class {{MetadataDef}}:
{code}
buff.append(" private final org.apache.calcite.rel.metadata.MetadataDef
def;\n");
for (Map.Entry, String> handlerAndName :
handlerToName.entrySet()) {
...
.append(" org.apache.calcite.rel.metadata.MetadataDef def");
{code}
This can lead to issues (e.g. if a downstream project shades Calcite library).
The safer way to do this is using {{Class#getName}}, as it is already done in
the rest of the code, e.g. in {{JaninoRelMetadataProvider:158}}:
{code}
.append(MetadataDef.class.getName())
{code}
was:
This seems caused by CALCITE-4546.
{{JaninoRelMetadataProvider}} lines 135 & 141 is using (for the generated code)
the hardcoded name of the class {{MetadataDef}}:
{code}
buff.append(" private final org.apache.calcite.rel.metadata.MetadataDef
def;\n");
for (Map.Entry, String> handlerAndName :
handlerToName.entrySet()) {
...
.append(" org.apache.calcite.rel.metadata.MetadataDef def");
{code}
This can lead to issues (e.g. if a downstream project shades Calcite library).
The safer way to do this is using {{Class#getName}}, as it is already done in
the rest of the code, e.g. in {{JaninoRelMetadataProvider:158}}:
{code}
.append(MetadataDef.class.getName())
{code}
> JaninoRelMetadataProvider uses hardcoded class name
> ---
>
> Key: CALCITE-4834
> URL: https://issues.apache.org/jira/browse/CALCITE-4834
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Fix For: 1.28.0
>
>
> This seems caused by CALCITE-4546
> ([fbbdf46|https://github.com/apache/calcite/commit/fbbdf465df46b4e6f9863d7d1dfdcb19a43f2032])
> {{JaninoRelMetadataProvider}} lines 135 & 141 is using (for the generated
> code) the hardcoded name of the class {{MetadataDef}}:
> {code}
> buff.append(" private final org.apache.calcite.rel.metadata.MetadataDef
> def;\n");
> for (Map.Entry, String> handlerAndName :
> handlerToName.entrySet()) {
> ...
> .append(" org.apache.calcite.rel.metadata.MetadataDef def");
> {code}
> This can lead to issues (e.g. if a downstream project shades Calcite
> library). The safer way to do this is using {{Class#getName}}, as it is
> already done in the rest of the code, e.g. in
> {{JaninoRelMetadataProvider:158}}:
> {code}
> .append(MetadataDef.class.getName())
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4834) JaninoRelMetadataProvider uses hardcoded class name
Ruben Q L created CALCITE-4834:
--
Summary: JaninoRelMetadataProvider uses hardcoded class name
Key: CALCITE-4834
URL: https://issues.apache.org/jira/browse/CALCITE-4834
Project: Calcite
Issue Type: Bug
Components: core
Reporter: Ruben Q L
Assignee: Ruben Q L
Fix For: 1.28.0
This seems caused by CALCITE-4546.
{{JaninoRelMetadataProvider}} lines 135 & 141 is using (for the generated code)
the hardcoded name of the class {{MetadataDef}}:
{code}
buff.append(" private final org.apache.calcite.rel.metadata.MetadataDef
def;\n");
for (Map.Entry, String> handlerAndName :
handlerToName.entrySet()) {
...
.append(" org.apache.calcite.rel.metadata.MetadataDef def");
{code}
This can lead to issues (e.g. if a downstream project shades Calcite library).
The safer way to do this is using {{Class#getName}}, as it is already done in
the rest of the code, e.g. in {{JaninoRelMetadataProvider:158}}:
{code}
.append(MetadataDef.class.getName())
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4834) JaninoRelMetadataProvider uses hardcoded class name
[
https://issues.apache.org/jira/browse/CALCITE-4834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-4834.
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/50dbd12f14f330f57a321a2402a7dcb8124f5112
> JaninoRelMetadataProvider uses hardcoded class name
> ---
>
> Key: CALCITE-4834
> URL: https://issues.apache.org/jira/browse/CALCITE-4834
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This seems caused by CALCITE-4546
> ([fbbdf46|https://github.com/apache/calcite/commit/fbbdf465df46b4e6f9863d7d1dfdcb19a43f2032])
> {{JaninoRelMetadataProvider}} lines 135 & 141 is using (for the generated
> code) the hardcoded name of the class {{MetadataDef}}:
> {code}
> buff.append(" private final org.apache.calcite.rel.metadata.MetadataDef
> def;\n");
> for (Map.Entry, String> handlerAndName :
> handlerToName.entrySet()) {
> ...
> .append(" org.apache.calcite.rel.metadata.MetadataDef def");
> {code}
> This can lead to issues (e.g. if a downstream project shades Calcite
> library). The safer way to do this is using {{Class#getName}}, as it is
> already done in the rest of the code, e.g. in
> {{JaninoRelMetadataProvider:158}}:
> {code}
> .append(MetadataDef.class.getName())
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4777) Casting from DECIMAL to BOOLEAN throws an exception
[
https://issues.apache.org/jira/browse/CALCITE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425375#comment-17425375
]
Ruben Q L commented on CALCITE-4777:
Just to confirm and to avoid any confusion, with the proposed PR:
- Casting decimal (and float, real, double) into boolean will be forbidden (it
will fail with an appropriate error message, and casting compatibility matrix
in the site will be updated).
- Casting integer types into boolean is (and will continue being) allowed. A
different issue is that currently casting integers into boolean always returns
false (instead of zero->false; non-zero->true), but there is another ticket to
deal with this problem: CALCITE-4782.
Do we all agree this is the intended behavior that we want to set?
> Casting from DECIMAL to BOOLEAN throws an exception
> ---
>
> Key: CALCITE-4777
> URL: https://issues.apache.org/jira/browse/CALCITE-4777
> Project: Calcite
> Issue Type: Bug
>Reporter: xuyangzhong
>Assignee: xuyangzhong
>Priority: Major
> Labels: pull-request-available
> Attachments: calcite.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> My sql is the following:
> {code:java}
> // code placeholder
> select * from test where cast (0.10915913549909961 as boolean){code}
>
> I want to simplify the cast. An exception is thrown:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.lang.RuntimeException: while resolving method
> 'booleanValue' in class class java.math.BigDecimal at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:424) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:435) at
> org.apache.calcite.linq4j.tree.Expressions.unbox(Expressions.java:1453) at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:398)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:326)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateCast(RexToLixTranslator.java:543)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$CastImplementor.implementSafe(RexImpTable.java:2450)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.genValueStatement(RexImpTable.java:2894)
> at
> org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.implement(RexImpTable.java:2859)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:1089)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:174) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:975)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitLocalRef(RexToLixTranslator.java:90)
> at org.apache.calcite.rex.RexLocalRef.accept(RexLocalRef.java:75) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:237)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:231)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateList(RexToLixTranslator.java:823)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateProjects(RexToLixTranslator.java:198)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:90)
> at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:66) at
> org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:128) at
> org.apache.calcite.rex.RexSimplify.simplifyCast(RexSimplify.java:2101) at
> org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:326) at
> org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:287) at
> org.apache.flink.table.examples.java.tests.CalciteTest.main(CalciteTest.java:47)
> Caused by: java.lang.NoSuchMethodException:
> java.math.BigDecimal.booleanValue() at
> java.lang.Class.getMethod(Class.java:1786) at
> org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:421) ... 25
> more
> {code}
> In order to avoid that I used the wrong rule or it caused by my bad coding, i
> write the test case following:
>
> {code:java}
> // code placeholder
> JavaTypeFactory typeFactory = new
> JavaTypeFactoryImpl(RelDataTypeSystem.DEFAULT);
> RexBuilder rexBuilder = new RexBuilder(typeFactory);
> final RexSimplify simplify = new RexSimplify(rexBuilder,
> RelOptPredicateList.EMPTY, RexUtil.EXECUTOR);
> RelDataType type = new BasicSqlType(typeFactory.getTypeSystem(),
> SqlTypeName.BOOLEAN);
> RelDataType bb = new
> BasicSqlType(typeFactory.getTypeSystem(),SqlTypeName.DECIMAL,18,17);
> SqlOperator op = new SqlCastFunction();
>
[jira] [Commented] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17417985#comment-17417985
]
Ruben Q L commented on CALCITE-4773:
Comments have been improved, they include now an example of the rule's
application.
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4773) RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-4773.
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/f4db84c3dd1674b929fede53bbd563b9e7101c71
> RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row
> type
> --
>
> Key: CALCITE-4773
> URL: https://issues.apache.org/jira/browse/CALCITE-4773
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The problem can be reproduced with the following test (to be added to
> {{unnest.iq}}):
> {noformat}
> SELECT unnested_outer.val, COUNT(1) AS count_val
> FROM
> (
> SELECT *
> FROM (VALUES (1, array [3, 5]), (2, array [6, 4, 6]))
> AS u(x, y)
> ) AS t, UNNEST(t.y) AS unnested_outer(val)
> WHERE
> (
> SELECT COUNT(unnested_inner.val) > 0
> FROM UNNEST(t.y) AS unnested_inner(val)
> WHERE unnested_inner.val = 4
> )
> GROUP BY unnested_outer.val
> ORDER BY count_val DESC, unnested_outer.val ASC;
> +-+---+
> | VAL | COUNT_VAL |
> +-+---+
> | 6 | 2 |
> | 4 | 1 |
> +-+---+
> (2 rows)
> !ok
> {noformat}
> When the test is executed, it fails with the following error:
> {noformat}
> > java.lang.AssertionError: Cannot add expression of different type to set:
> > set type is RecordType(BOOLEAN NOT NULL $f0) NOT NULL
> > expression type is RecordType(BOOLEAN $f0) NOT NULL
> > set is
> > rel#24436:LogicalAggregate.NONE.[](input=HepRelVertex#24435,group={},agg#0=SINGLE_VALUE($0))
> > expression is LogicalProject($f0=[CAST(>($0, 0)):BOOLEAN])
> > LogicalAggregate(group=[{}], agg#0=[COUNT()])
> > LogicalFilter(condition=[=($0, 4)])
> > LogicalProject(VAL=[$0])
> > Uncollect
> > LogicalProject(Y=[$cor1.Y])
> > LogicalValues(tuples=[[{ 0 }]])
> 249a254,325
> > at
> > org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:391)
> > at
> > org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:269)
> > at
> > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:284)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1923)
> > at
> > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:341)
> > at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
> > at
> > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208)
> > at
> > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:195)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:378)
> > at
> > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:221)
> ...
> {noformat}
> The root cause seems to be in this piece of code inside {{RelDecorrelator}}'s
> {{RemoveSingleAggregateRule#onMatch}}:
> {code:java}
> // singleAggRel produces a nullable type, so create the new
> // projection that casts proj expr to a nullable type.
> final RelBuilder relBuilder = call.builder();
> final RelDataType type =
> relBuilder.getTypeFactory()
> .createTypeWithNullability(projExprs.get(0).getType(), true);
> final RexNode cast = relBuilder.getRexBuilder().makeCast(type,
> projExprs.get(0));
> relBuilder.push(aggregate).project(cast);
> call.transformTo(relBuilder.build());
> {code}
> Note that the comment assumes that _"singleAggRel produces a nullable type"_,
> but in this particular case, it seems to produce a non-nullable type, so
> probably this piece of code needs to be adapted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4737) Add RelOptPlanner visualizer for debugging
[ https://issues.apache.org/jira/browse/CALCITE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L resolved CALCITE-4737. Resolution: Fixed Fixed via https://github.com/apache/calcite/commit/ce2531148f0c9990792068b12000deac827fc831 Thanks [~thomas.rebele] & [~zuozhiw] for the PR, and everyone else involved in the review! > Add RelOptPlanner visualizer for debugging > -- > > Key: CALCITE-4737 > URL: https://issues.apache.org/jira/browse/CALCITE-4737 > Project: Calcite > Issue Type: New Feature >Reporter: Julian Hyde >Assignee: Thomas Rebele >Priority: Major > Labels: pull-request-available > Fix For: 1.29.0 > > Time Spent: 9h 20m > Remaining Estimate: 0h > > Add Volcano visualizer for debugging. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (CALCITE-4930) Update log4j2 version to 2.15.0
[ https://issues.apache.org/jira/browse/CALCITE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L updated CALCITE-4930: --- Fix Version/s: 1.29.0 > Update log4j2 version to 2.15.0 > --- > > Key: CALCITE-4930 > URL: https://issues.apache.org/jira/browse/CALCITE-4930 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: Ada Wong >Priority: Blocker > Labels: pull-request-available > Fix For: 1.29.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > 2.0 <= Apache log4j2 <= 2.14.1 have a RCE zero day. > [https://www.cyberkendra.com/2021/12/worst-log4j-rce-zeroday-dropped-on.html] > [https://www.lunasec.io/docs/blog/log4j-zero-day/] > h4. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (CALCITE-4737) Add Volcano visualizer for debugging
[
https://issues.apache.org/jira/browse/CALCITE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460648#comment-17460648
]
Ruben Q L commented on CALCITE-4737:
In {{org/apache/calcite/config/CalciteSystemProperty.java}} there seems to be
already some properties that control what to be displayed by Volcano planner:
{code}
/**
* Whether to include a GraphViz representation when dumping the state of the
Volcano planner.
*/
public static final CalciteSystemProperty DUMP_GRAPHVIZ =
booleanProperty("calcite.volcano.dump.graphviz", true);
/**
* Whether to include RelSet information when dumping the state
of the Volcano planner.
*/
public static final CalciteSystemProperty DUMP_SETS =
booleanProperty("calcite.volcano.dump.sets", true);
{code}
These properties are used by {{VolcanoPlanner#dump}} (called when
{{LOGGER.isTraceEnabled()}} or in case of a {{CannotPlanException}}).
What about adding a new property in {{CalciteSystemProperty.java}}, e.g.
something like:
{code}
public static final CalciteSystemProperty DUMP_HTMLGRAPHVIZ =
booleanProperty("calcite.volcano.dump.htmlgraphviz", false);
{code}
Together with other necessary properties (e.g. default output directory,
filename prefix, etc). And take it into consideration in
{{VolcanoPlanner#dump}}. Then Calcite clients could disable the "traditional"
graphviz and enable the new visualizer (or enable both, or neither) as they
wish.
What do you think, [~thomas.rebele] et al?
> Add Volcano visualizer for debugging
>
>
> Key: CALCITE-4737
> URL: https://issues.apache.org/jira/browse/CALCITE-4737
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Thomas Rebele
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.29.0
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Add Volcano visualizer for debugging.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CALCITE-4737) Add Volcano visualizer for debugging
[ https://issues.apache.org/jira/browse/CALCITE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460721#comment-17460721 ] Ruben Q L commented on CALCITE-4737: [~zabetak], I agree to postpone the property discussion into a follow-up ticket. I think the feature as it is (with some final details to be fixed in the PR) will be a valuable addition for Calcite 1.29. > Add Volcano visualizer for debugging > > > Key: CALCITE-4737 > URL: https://issues.apache.org/jira/browse/CALCITE-4737 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Thomas Rebele >Priority: Major > Labels: pull-request-available > Fix For: 1.29.0 > > Time Spent: 7h 50m > Remaining Estimate: 0h > > Add Volcano visualizer for debugging. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (CALCITE-4941) SemiJoinRule loses hints
Ruben Q L created CALCITE-4941: -- Summary: SemiJoinRule loses hints Key: CALCITE-4941 URL: https://issues.apache.org/jira/browse/CALCITE-4941 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.28.0 Reporter: Ruben Q L Assignee: Ruben Q L Fix For: 1.29.0 SemiJoinRule does not propagate the (potential) hints from the input Join to the output SemiJoin -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (CALCITE-4941) SemiJoinRule loses hints
[ https://issues.apache.org/jira/browse/CALCITE-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L resolved CALCITE-4941. Resolution: Fixed Fixed via https://github.com/apache/calcite/commit/7d342b3eac92b34a9196a5486e4ebe172d843f2b > SemiJoinRule loses hints > > > Key: CALCITE-4941 > URL: https://issues.apache.org/jira/browse/CALCITE-4941 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: Ruben Q L >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Fix For: 1.29.0 > > Time Spent: 1h > Remaining Estimate: 0h > > SemiJoinRule does not propagate the (potential) hints from the input Join to > the output SemiJoin -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (CALCITE-4054) RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
[
https://issues.apache.org/jira/browse/CALCITE-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L reassigned CALCITE-4054:
--
Assignee: Ruben Q L
> RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
> -
>
> Key: CALCITE-4054
> URL: https://issues.apache.org/jira/browse/CALCITE-4054
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Minor
>
> This is not a very urgent problem, since this is an exception on an operator
> tagged as "Experimental", and several "not-so-usual" circumstances must occur
> in order to reach the NPE:
> - We need a "recursive plan", i.e. a RepeatUnion with a TransientScan (see
> examples in EnumerableRepeatUnionTest.java).
> - Inside the repeat union, there must be a Correlate, or a Join that gets
> implemented as a Correlate due to JoinToCorrelateRule.
> - We have the TransientScan (i.e. the scan on the transient table) on the
> right-hand-side of the correlate/join. Normally, transientScan always appear
> on the LHS of a correlate/join (because when the plan is built they need the
> top of the RelBuilder's stack to get its rowType); however, it can get on the
> RHS due to JoinCommuteRule.
> We can force these conditions and reproduce the issue with the following unit
> test (in EnumerableRepeatUnionTest.java):
> {code:java}
> @Test void testRepeatUnionWithCorrelateWithTransientScanOnItsRight() {
> CalciteAssert.that()
> .with(CalciteConnectionProperty.LEX, Lex.JAVA)
> .with(CalciteConnectionProperty.FORCE_DECORRELATE, false)
> .withSchema("s", new ReflectiveSchema(new HierarchySchema()))
> .query("?")
> .withHook(Hook.PLANNER, (Consumer) planner -> {
> planner.addRule(JoinToCorrelateRule.INSTANCE);
> planner.removeRule(JoinCommuteRule.INSTANCE);
> planner.removeRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
> planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
> })
> .withRel(builder -> {
> builder
> // WITH RECURSIVE delta(empid, name) as (
> // SELECT empid, name FROM emps WHERE empid = 2
> // UNION ALL
> // SELECT e.empid, e.name FROM delta d
> //JOIN hierarchies h ON d.empid =
> h.managerid
> //JOIN emps eON
> h.subordinateid = e.empid
> // )
> // SELECT empid, name FROM delta
> .scan("s", "emps")
> .filter(
> builder.equals(
> builder.field("empid"),
> builder.literal(2)))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .transientScan("#DELTA#");
> RelNode transientScan = builder.build(); // pop the transientScan
> to use it later
> builder
> .scan("s", "hierarchies")
> .push(transientScan) // use the transientScan as right input of
> the join
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "#DELTA#", "empid"),
> builder.field(2, "hierarchies", "managerid")))
> .scan("s", "emps")
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "hierarchies", "subordinateid"),
> builder.field(2, "emps", "empid")))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .repeatUnion("#DELTA#", true);
> return builder.build();
> })
> .explainHookMatches(""
> + "EnumerableRepeatUnion(all=[true])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..4=[{inputs}], expr#5=[2],
> expr#6=[=($t0, $t5)], empid=[$t0], name=[$t2], $condition=[$t6])\n"
> + " EnumerableTableScan(table=[[s, emps]])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..8=[{inputs}], empid=[$t4],
> name=[$t6])\n"
> + " EnumerableCorrelate(correlation=[$cor1],
> joinType=[inner], requiredColumns=[{1}])\n"
> // It is important to have EnumerableCorrelate + #DELTA# table
> scan on its right
> + "
[jira] [Updated] (CALCITE-4054) RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
[
https://issues.apache.org/jira/browse/CALCITE-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4054:
---
Fix Version/s: 1.30.0
> RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
> -
>
> Key: CALCITE-4054
> URL: https://issues.apache.org/jira/browse/CALCITE-4054
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Minor
> Fix For: 1.30.0
>
>
> This is not a very urgent problem, since this is an exception on an operator
> tagged as "Experimental", and several "not-so-usual" circumstances must occur
> in order to reach the NPE:
> - We need a "recursive plan", i.e. a RepeatUnion with a TransientScan (see
> examples in EnumerableRepeatUnionTest.java).
> - Inside the repeat union, there must be a Correlate, or a Join that gets
> implemented as a Correlate due to JoinToCorrelateRule.
> - We have the TransientScan (i.e. the scan on the transient table) on the
> right-hand-side of the correlate/join. Normally, transientScan always appear
> on the LHS of a correlate/join (because when the plan is built they need the
> top of the RelBuilder's stack to get its rowType); however, it can get on the
> RHS due to JoinCommuteRule.
> We can force these conditions and reproduce the issue with the following unit
> test (in EnumerableRepeatUnionTest.java):
> {code:java}
> @Test void testRepeatUnionWithCorrelateWithTransientScanOnItsRight() {
> CalciteAssert.that()
> .with(CalciteConnectionProperty.LEX, Lex.JAVA)
> .with(CalciteConnectionProperty.FORCE_DECORRELATE, false)
> .withSchema("s", new ReflectiveSchema(new HierarchySchema()))
> .query("?")
> .withHook(Hook.PLANNER, (Consumer) planner -> {
> planner.addRule(JoinToCorrelateRule.INSTANCE);
> planner.removeRule(JoinCommuteRule.INSTANCE);
> planner.removeRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
> planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
> })
> .withRel(builder -> {
> builder
> // WITH RECURSIVE delta(empid, name) as (
> // SELECT empid, name FROM emps WHERE empid = 2
> // UNION ALL
> // SELECT e.empid, e.name FROM delta d
> //JOIN hierarchies h ON d.empid =
> h.managerid
> //JOIN emps eON
> h.subordinateid = e.empid
> // )
> // SELECT empid, name FROM delta
> .scan("s", "emps")
> .filter(
> builder.equals(
> builder.field("empid"),
> builder.literal(2)))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .transientScan("#DELTA#");
> RelNode transientScan = builder.build(); // pop the transientScan
> to use it later
> builder
> .scan("s", "hierarchies")
> .push(transientScan) // use the transientScan as right input of
> the join
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "#DELTA#", "empid"),
> builder.field(2, "hierarchies", "managerid")))
> .scan("s", "emps")
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "hierarchies", "subordinateid"),
> builder.field(2, "emps", "empid")))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .repeatUnion("#DELTA#", true);
> return builder.build();
> })
> .explainHookMatches(""
> + "EnumerableRepeatUnion(all=[true])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..4=[{inputs}], expr#5=[2],
> expr#6=[=($t0, $t5)], empid=[$t0], name=[$t2], $condition=[$t6])\n"
> + " EnumerableTableScan(table=[[s, emps]])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..8=[{inputs}], empid=[$t4],
> name=[$t6])\n"
> + " EnumerableCorrelate(correlation=[$cor1],
> joinType=[inner], requiredColumns=[{1}])\n"
> // It is important to have EnumerableCorrelate + #DELTA# table
> scan on its right
>
[jira] [Assigned] (CALCITE-3673) ListTransientTable should not leave tables in the schema
[ https://issues.apache.org/jira/browse/CALCITE-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L reassigned CALCITE-3673: -- Assignee: Ruben Q L > ListTransientTable should not leave tables in the schema > > > Key: CALCITE-3673 > URL: https://issues.apache.org/jira/browse/CALCITE-3673 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.21.0 >Reporter: Vladimir Sitnikov >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > 1) org.apache.calcite.schema.impl.ListTransientTable#scan adds itself to a > schema: root.getRootSchema().add(name, this); however, it misses cleanup, so > it looks like a memory leak > 2) ListTransientTable is not implementable with EnumerableTableScan as > org.apache.calcite.tools.RelBuilder#transientScan(java.lang.String, > org.apache.calcite.rel.type.RelDataType) is using > org.apache.calcite.prepare.RelOptTableImpl#create(org.apache.calcite.plan.RelOptSchema, > org.apache.calcite.rel.type.RelDataType, org.apache.calcite.schema.Table, > com.google.common.collect.ImmutableList) which creates > RelOptTableImpl with NULL expressionFunction which means > org.apache.calcite.prepare.RelOptTableImpl#getExpression returns null -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (CALCITE-3673) ListTransientTable should not leave tables in the schema
[ https://issues.apache.org/jira/browse/CALCITE-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L updated CALCITE-3673: --- Fix Version/s: 1.30.0 > ListTransientTable should not leave tables in the schema > > > Key: CALCITE-3673 > URL: https://issues.apache.org/jira/browse/CALCITE-3673 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.21.0 >Reporter: Vladimir Sitnikov >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Fix For: 1.30.0 > > Time Spent: 10m > Remaining Estimate: 0h > > 1) org.apache.calcite.schema.impl.ListTransientTable#scan adds itself to a > schema: root.getRootSchema().add(name, this); however, it misses cleanup, so > it looks like a memory leak > 2) ListTransientTable is not implementable with EnumerableTableScan as > org.apache.calcite.tools.RelBuilder#transientScan(java.lang.String, > org.apache.calcite.rel.type.RelDataType) is using > org.apache.calcite.prepare.RelOptTableImpl#create(org.apache.calcite.plan.RelOptSchema, > org.apache.calcite.rel.type.RelDataType, org.apache.calcite.schema.Table, > com.google.common.collect.ImmutableList) which creates > RelOptTableImpl with NULL expressionFunction which means > org.apache.calcite.prepare.RelOptTableImpl#getExpression returns null -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (CALCITE-4054) RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
[
https://issues.apache.org/jira/browse/CALCITE-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-4054:
---
Labels: pull-request-available (was: )
> RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
> -
>
> Key: CALCITE-4054
> URL: https://issues.apache.org/jira/browse/CALCITE-4054
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.30.0
>
>
> This is not a very urgent problem, since this is an exception on an operator
> tagged as "Experimental", and several "not-so-usual" circumstances must occur
> in order to reach the NPE:
> - We need a "recursive plan", i.e. a RepeatUnion with a TransientScan (see
> examples in EnumerableRepeatUnionTest.java).
> - Inside the repeat union, there must be a Correlate, or a Join that gets
> implemented as a Correlate due to JoinToCorrelateRule.
> - We have the TransientScan (i.e. the scan on the transient table) on the
> right-hand-side of the correlate/join. Normally, transientScan always appear
> on the LHS of a correlate/join (because when the plan is built they need the
> top of the RelBuilder's stack to get its rowType); however, it can get on the
> RHS due to JoinCommuteRule.
> We can force these conditions and reproduce the issue with the following unit
> test (in EnumerableRepeatUnionTest.java):
> {code:java}
> @Test void testRepeatUnionWithCorrelateWithTransientScanOnItsRight() {
> CalciteAssert.that()
> .with(CalciteConnectionProperty.LEX, Lex.JAVA)
> .with(CalciteConnectionProperty.FORCE_DECORRELATE, false)
> .withSchema("s", new ReflectiveSchema(new HierarchySchema()))
> .query("?")
> .withHook(Hook.PLANNER, (Consumer) planner -> {
> planner.addRule(JoinToCorrelateRule.INSTANCE);
> planner.removeRule(JoinCommuteRule.INSTANCE);
> planner.removeRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
> planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
> })
> .withRel(builder -> {
> builder
> // WITH RECURSIVE delta(empid, name) as (
> // SELECT empid, name FROM emps WHERE empid = 2
> // UNION ALL
> // SELECT e.empid, e.name FROM delta d
> //JOIN hierarchies h ON d.empid =
> h.managerid
> //JOIN emps eON
> h.subordinateid = e.empid
> // )
> // SELECT empid, name FROM delta
> .scan("s", "emps")
> .filter(
> builder.equals(
> builder.field("empid"),
> builder.literal(2)))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .transientScan("#DELTA#");
> RelNode transientScan = builder.build(); // pop the transientScan
> to use it later
> builder
> .scan("s", "hierarchies")
> .push(transientScan) // use the transientScan as right input of
> the join
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "#DELTA#", "empid"),
> builder.field(2, "hierarchies", "managerid")))
> .scan("s", "emps")
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "hierarchies", "subordinateid"),
> builder.field(2, "emps", "empid")))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .repeatUnion("#DELTA#", true);
> return builder.build();
> })
> .explainHookMatches(""
> + "EnumerableRepeatUnion(all=[true])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..4=[{inputs}], expr#5=[2],
> expr#6=[=($t0, $t5)], empid=[$t0], name=[$t2], $condition=[$t6])\n"
> + " EnumerableTableScan(table=[[s, emps]])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..8=[{inputs}], empid=[$t4],
> name=[$t6])\n"
> + " EnumerableCorrelate(correlation=[$cor1],
> joinType=[inner], requiredColumns=[{1}])\n"
> // It is important to have
[jira] [Commented] (CALCITE-4999) `least restrictive` type inference does not work when one operand is scalar and another is row with one field
[
https://issues.apache.org/jira/browse/CALCITE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492473#comment-17492473
]
Ruben Q L commented on CALCITE-4999:
If I an not mistaken, the current Calcite implementation seems more powerful
(or more flexible) that e.g. the BigQuery one, because the ARRAY subquery
allows multiple columns.
For instance a single column ARRAY:
{code:sql}
SELECT r.r_name, ARRAY(
SELECT n.n_name FROM nation n WHERE n.n_regionkey = r.r_regionkey
)
FROM region r ORDER BY r.r_name
{code}
Returns a record type with one field for the ARRAY subquery (which seemed fine
so far, but it could potentially be a scalar: the point of this discussion).
But it also supports the case of a multi-column ARRAY:
{code:sql}
SELECT r.r_name, ARRAY(
SELECT n.n_nationkey, n.n_name FROM nation n WHERE n.n_regionkey =
r.r_regionkey
)
FROM region r ORDER BY r.r_name
{code}
It returns a record type with two fields for the ARRAY subquery (which is
perfectly fine).
If we changed the first case to return a scalar, wouldn't it be a misalignment
compared to the second? (not sure, just raising the question)
> `least restrictive` type inference does not work when one operand is scalar
> and another is row with one field
> -
>
> Key: CALCITE-4999
> URL: https://issues.apache.org/jira/browse/CALCITE-4999
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Dmitry Sysolyatin
>Assignee: Dmitry Sysolyatin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I faced with a issue that the following query:
> {code:java}
> SELECT ARRAY_CONCAT(ARRAY['1', '2'], array(select 'toast.' || x from
> unnest(ARRAY['1','2']) x)) {code}
> didn't work, because of:
> {code:java}
> java.lang.IllegalArgumentException: Cannot infer return type for
> ARRAY_CONCAT; operand types: [CHAR(1) ARRAY, RecordType(CHAR(7) EXPR$0)
> ARRAY]{code}
> `least restrictive` type inference does not work when one operand is scalar
> and another is row with one field
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CALCITE-4995) AssertionError caused by RelFieldTrimmer on SEMI/ANTI join
[
https://issues.apache.org/jira/browse/CALCITE-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492710#comment-17492710
]
Ruben Q L commented on CALCITE-4995:
Fixed via
https://github.com/apache/calcite/commit/04029ff5a1918c373dced4c62750fa7093b311ab
> AssertionError caused by RelFieldTrimmer on SEMI/ANTI join
> --
>
> Key: CALCITE-4995
> URL: https://issues.apache.org/jira/browse/CALCITE-4995
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.29.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.30.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> It seems {{RelFieldTrimmer}} can cause an {{AssertionError}} (or later an
> {{ArrayIndexOutOfBoundsException}} if assertions are disabled) on certain
> plans involving SEMI/ANTI join (i.e. joins that do NOT project the RHS
> fields).
> The root cause seems to be the "early return" in
> {{RelFieldTrimmer#trimFields(Join join, ImmutableBitSet fieldsUsed,
> Set extraFields)}} when nothing has been trimmed inside
> join's inputs (so the join itself can be return as it is):
> {code:java}
> if (changeCount == 0
> && mapping.isIdentity()) {
> return result(join, Mappings.createIdentity(fieldCount));
> }
> {code}
> The problem is that this {{fieldCount}} is an addition of LHS + RHS fields (+
> system fields); but in case of a SEMI/ANTI the mappings to be returned must
> not consider RHS fields (since they are not projected by these join types).
> The problem only happens here (when the trimmer does not trim the join).
> Notice that, a few lines below, in the "other return scenario" of the method
> (when something has been trimmed), there is a special treatment of the
> mapping for ANTI/SEMI, so things will work fine in this case:
> {code:java}
> switch (join.getJoinType()) {
> case SEMI:
> case ANTI:
> // For SemiJoins and AntiJoins only map fields from the left-side
> if (join.getJoinType() == JoinRelType.SEMI) {
> relBuilder.semiJoin(newConditionExpr);
> } else {
> relBuilder.antiJoin(newConditionExpr);
> }
> Mapping inputMapping = inputMappings.get(0);
> mapping = Mappings.create(MappingType.INVERSE_SURJECTION,
> join.getRowType().getFieldCount(),
> newSystemFieldCount + inputMapping.getTargetCount());
> for (int i = 0; i < newSystemFieldCount; ++i) {
> mapping.set(i, i);
> }
> offset = systemFieldCount;
> newOffset = newSystemFieldCount;
> for (IntPair pair : inputMapping) {
> mapping.set(pair.source + offset, pair.target + newOffset);
> }
> break;
> default:
> relBuilder.join(join.getJoinType(), newConditionExpr);
> }
> relBuilder.hints(join.getHints());
> return result(relBuilder.build(), mapping);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (CALCITE-4995) AssertionError caused by RelFieldTrimmer on SEMI/ANTI join
[
https://issues.apache.org/jira/browse/CALCITE-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-4995.
Resolution: Fixed
> AssertionError caused by RelFieldTrimmer on SEMI/ANTI join
> --
>
> Key: CALCITE-4995
> URL: https://issues.apache.org/jira/browse/CALCITE-4995
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.29.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.30.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> It seems {{RelFieldTrimmer}} can cause an {{AssertionError}} (or later an
> {{ArrayIndexOutOfBoundsException}} if assertions are disabled) on certain
> plans involving SEMI/ANTI join (i.e. joins that do NOT project the RHS
> fields).
> The root cause seems to be the "early return" in
> {{RelFieldTrimmer#trimFields(Join join, ImmutableBitSet fieldsUsed,
> Set extraFields)}} when nothing has been trimmed inside
> join's inputs (so the join itself can be return as it is):
> {code:java}
> if (changeCount == 0
> && mapping.isIdentity()) {
> return result(join, Mappings.createIdentity(fieldCount));
> }
> {code}
> The problem is that this {{fieldCount}} is an addition of LHS + RHS fields (+
> system fields); but in case of a SEMI/ANTI the mappings to be returned must
> not consider RHS fields (since they are not projected by these join types).
> The problem only happens here (when the trimmer does not trim the join).
> Notice that, a few lines below, in the "other return scenario" of the method
> (when something has been trimmed), there is a special treatment of the
> mapping for ANTI/SEMI, so things will work fine in this case:
> {code:java}
> switch (join.getJoinType()) {
> case SEMI:
> case ANTI:
> // For SemiJoins and AntiJoins only map fields from the left-side
> if (join.getJoinType() == JoinRelType.SEMI) {
> relBuilder.semiJoin(newConditionExpr);
> } else {
> relBuilder.antiJoin(newConditionExpr);
> }
> Mapping inputMapping = inputMappings.get(0);
> mapping = Mappings.create(MappingType.INVERSE_SURJECTION,
> join.getRowType().getFieldCount(),
> newSystemFieldCount + inputMapping.getTargetCount());
> for (int i = 0; i < newSystemFieldCount; ++i) {
> mapping.set(i, i);
> }
> offset = systemFieldCount;
> newOffset = newSystemFieldCount;
> for (IntPair pair : inputMapping) {
> mapping.set(pair.source + offset, pair.target + newOffset);
> }
> break;
> default:
> relBuilder.join(join.getJoinType(), newConditionExpr);
> }
> relBuilder.hints(join.getHints());
> return result(relBuilder.build(), mapping);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CALCITE-5048) Query crashes `not a literal` for dynamic parameter
[
https://issues.apache.org/jira/browse/CALCITE-5048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17507692#comment-17507692
]
Ruben Q L commented on CALCITE-5048:
If I am not mistaken, there are parts of the code inside Calcite which only
support literals (and not dynamic parameters) for LIMIT (and OFFSET).
RelDecorrelator seems to be one of them (but there might be others).
> Query crashes `not a literal` for dynamic parameter
> ---
>
> Key: CALCITE-5048
> URL: https://issues.apache.org/jira/browse/CALCITE-5048
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.29.0
>Reporter: Corvin Kuebler
>Priority: Major
>
> This query:
> {code:java}
> // Does not run
> @Test void testDynamicParameterLimit() {
> CalciteAssert.hr()
> .query("SELECT * FROM \"hr\".\"emps\" AS a " +
> "WHERE \"deptno\" = " +
> "(SELECT MAX(\"deptno\") " +
> "FROM \"hr\".\"emps\" AS b " +
> "WHERE a.\"empid\" = b.\"empid\"" +
> ") LIMIT ?")
> .runs();
> }{code}
> results in the following exception:
> {code:java}
> not a literal: ?0
> java.lang.AssertionError: not a literal: ?0
> at org.apache.calcite.rex.RexLiteral.findValue(RexLiteral.java:1208)
> at org.apache.calcite.rex.RexLiteral.intValue(RexLiteral.java:1183)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:481)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:711)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:306)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:230)
> at
> org.apache.calcite.tools.Programs$DecorrelateProgram.run(Programs.java:361)
> at
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:336)
> at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:177)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:312)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:220){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CALCITE-4999) `least restrictive` type inference does not work when one operand is scalar and another is row with one field
[
https://issues.apache.org/jira/browse/CALCITE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493206#comment-17493206
]
Ruben Q L commented on CALCITE-4999:
[~dmsysolyatin] you can find some draft versions of the standard in Wikipedia:
https://en.wikipedia.org/wiki/SQL#SQL_standards_documents
I have checked the standard (relevant parts: and
); and it would seem that, indeed, Calcite is
not following it:
{noformat}
6.38
Function
Specify construction of an array.
Format
::=
|
[...]
::=
ARRAY
Syntax Rules
[...]
3) If is specified, then
a) The QE simply contained in the shall be
of degree 1 (one). Let ET be the declared type of the column in the result of
.
b) The declared type of the is array with
element type ET and maximum cardinality equal to the implementation-defined
maximum cardinality IMDC for such array types.
{noformat}
> `least restrictive` type inference does not work when one operand is scalar
> and another is row with one field
> -
>
> Key: CALCITE-4999
> URL: https://issues.apache.org/jira/browse/CALCITE-4999
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Dmitry Sysolyatin
>Assignee: Dmitry Sysolyatin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I faced with a issue that the following query:
> {code:java}
> SELECT ARRAY_CONCAT(ARRAY['1', '2'], array(select 'toast.' || x from
> unnest(ARRAY['1','2']) x)) {code}
> didn't work, because of:
> {code:java}
> java.lang.IllegalArgumentException: Cannot infer return type for
> ARRAY_CONCAT; operand types: [CHAR(1) ARRAY, RecordType(CHAR(7) EXPR$0)
> ARRAY]{code}
> `least restrictive` type inference does not work when one operand is scalar
> and another is row with one field
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CALCITE-4999) `least restrictive` type inference does not work when one operand is scalar and another is row with one field
[
https://issues.apache.org/jira/browse/CALCITE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493757#comment-17493757
]
Ruben Q L commented on CALCITE-4999:
bq. So I think the solution is to return an array of scalars (e.g. INTEGER
ARRAY) when the query returns 1 column, and continue to return an array of ROW
when the query has 2 or more columns. And do the same for MULTISET.
+1
> `least restrictive` type inference does not work when one operand is scalar
> and another is row with one field
> -
>
> Key: CALCITE-4999
> URL: https://issues.apache.org/jira/browse/CALCITE-4999
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Dmitry Sysolyatin
>Assignee: Dmitry Sysolyatin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I faced with a issue that the following query:
> {code:java}
> SELECT ARRAY_CONCAT(ARRAY['1', '2'], array(select 'toast.' || x from
> unnest(ARRAY['1','2']) x)) {code}
> didn't work, because of:
> {code:java}
> java.lang.IllegalArgumentException: Cannot infer return type for
> ARRAY_CONCAT; operand types: [CHAR(1) ARRAY, RecordType(CHAR(7) EXPR$0)
> ARRAY]{code}
> `least restrictive` type inference does not work when one operand is scalar
> and another is row with one field
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CALCITE-3673) ListTransientTable should not leave tables in the schema
[ https://issues.apache.org/jira/browse/CALCITE-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494660#comment-17494660 ] Ruben Q L commented on CALCITE-3673: Fixed via https://github.com/apache/calcite/commit/9c4f3bb540dd67a0ffefc09f4ebd98d2be65bb14 > ListTransientTable should not leave tables in the schema > > > Key: CALCITE-3673 > URL: https://issues.apache.org/jira/browse/CALCITE-3673 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.21.0 >Reporter: Vladimir Sitnikov >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Fix For: 1.30.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > 1) org.apache.calcite.schema.impl.ListTransientTable#scan adds itself to a > schema: root.getRootSchema().add(name, this); however, it misses cleanup, so > it looks like a memory leak > 2) ListTransientTable is not implementable with EnumerableTableScan as > org.apache.calcite.tools.RelBuilder#transientScan(java.lang.String, > org.apache.calcite.rel.type.RelDataType) is using > org.apache.calcite.prepare.RelOptTableImpl#create(org.apache.calcite.plan.RelOptSchema, > org.apache.calcite.rel.type.RelDataType, org.apache.calcite.schema.Table, > com.google.common.collect.ImmutableList) which creates > RelOptTableImpl with NULL expressionFunction which means > org.apache.calcite.prepare.RelOptTableImpl#getExpression returns null -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (CALCITE-4054) RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
[
https://issues.apache.org/jira/browse/CALCITE-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494661#comment-17494661
]
Ruben Q L commented on CALCITE-4054:
Fixed via
https://github.com/apache/calcite/commit/9c4f3bb540dd67a0ffefc09f4ebd98d2be65bb14
> RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
> -
>
> Key: CALCITE-4054
> URL: https://issues.apache.org/jira/browse/CALCITE-4054
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.30.0
>
>
> This is not a very urgent problem, since this is an exception on an operator
> tagged as "Experimental", and several "not-so-usual" circumstances must occur
> in order to reach the NPE:
> - We need a "recursive plan", i.e. a RepeatUnion with a TransientScan (see
> examples in EnumerableRepeatUnionTest.java).
> - Inside the repeat union, there must be a Correlate, or a Join that gets
> implemented as a Correlate due to JoinToCorrelateRule.
> - We have the TransientScan (i.e. the scan on the transient table) on the
> right-hand-side of the correlate/join. Normally, transientScan always appear
> on the LHS of a correlate/join (because when the plan is built they need the
> top of the RelBuilder's stack to get its rowType); however, it can get on the
> RHS due to JoinCommuteRule.
> We can force these conditions and reproduce the issue with the following unit
> test (in EnumerableRepeatUnionTest.java):
> {code:java}
> @Test void testRepeatUnionWithCorrelateWithTransientScanOnItsRight() {
> CalciteAssert.that()
> .with(CalciteConnectionProperty.LEX, Lex.JAVA)
> .with(CalciteConnectionProperty.FORCE_DECORRELATE, false)
> .withSchema("s", new ReflectiveSchema(new HierarchySchema()))
> .query("?")
> .withHook(Hook.PLANNER, (Consumer) planner -> {
> planner.addRule(JoinToCorrelateRule.INSTANCE);
> planner.removeRule(JoinCommuteRule.INSTANCE);
> planner.removeRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
> planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
> })
> .withRel(builder -> {
> builder
> // WITH RECURSIVE delta(empid, name) as (
> // SELECT empid, name FROM emps WHERE empid = 2
> // UNION ALL
> // SELECT e.empid, e.name FROM delta d
> //JOIN hierarchies h ON d.empid =
> h.managerid
> //JOIN emps eON
> h.subordinateid = e.empid
> // )
> // SELECT empid, name FROM delta
> .scan("s", "emps")
> .filter(
> builder.equals(
> builder.field("empid"),
> builder.literal(2)))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .transientScan("#DELTA#");
> RelNode transientScan = builder.build(); // pop the transientScan
> to use it later
> builder
> .scan("s", "hierarchies")
> .push(transientScan) // use the transientScan as right input of
> the join
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "#DELTA#", "empid"),
> builder.field(2, "hierarchies", "managerid")))
> .scan("s", "emps")
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "hierarchies", "subordinateid"),
> builder.field(2, "emps", "empid")))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .repeatUnion("#DELTA#", true);
> return builder.build();
> })
> .explainHookMatches(""
> + "EnumerableRepeatUnion(all=[true])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..4=[{inputs}], expr#5=[2],
> expr#6=[=($t0, $t5)], empid=[$t0], name=[$t2], $condition=[$t6])\n"
> + " EnumerableTableScan(table=[[s, emps]])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..8=[{inputs}], empid=[$t4],
> name=[$t6])\n"
> + "
[jira] [Resolved] (CALCITE-3673) ListTransientTable should not leave tables in the schema
[ https://issues.apache.org/jira/browse/CALCITE-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L resolved CALCITE-3673. Resolution: Fixed > ListTransientTable should not leave tables in the schema > > > Key: CALCITE-3673 > URL: https://issues.apache.org/jira/browse/CALCITE-3673 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.21.0 >Reporter: Vladimir Sitnikov >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Fix For: 1.30.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > 1) org.apache.calcite.schema.impl.ListTransientTable#scan adds itself to a > schema: root.getRootSchema().add(name, this); however, it misses cleanup, so > it looks like a memory leak > 2) ListTransientTable is not implementable with EnumerableTableScan as > org.apache.calcite.tools.RelBuilder#transientScan(java.lang.String, > org.apache.calcite.rel.type.RelDataType) is using > org.apache.calcite.prepare.RelOptTableImpl#create(org.apache.calcite.plan.RelOptSchema, > org.apache.calcite.rel.type.RelDataType, org.apache.calcite.schema.Table, > com.google.common.collect.ImmutableList) which creates > RelOptTableImpl with NULL expressionFunction which means > org.apache.calcite.prepare.RelOptTableImpl#getExpression returns null -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (CALCITE-4054) RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
[
https://issues.apache.org/jira/browse/CALCITE-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-4054.
Resolution: Fixed
> RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
> -
>
> Key: CALCITE-4054
> URL: https://issues.apache.org/jira/browse/CALCITE-4054
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.30.0
>
>
> This is not a very urgent problem, since this is an exception on an operator
> tagged as "Experimental", and several "not-so-usual" circumstances must occur
> in order to reach the NPE:
> - We need a "recursive plan", i.e. a RepeatUnion with a TransientScan (see
> examples in EnumerableRepeatUnionTest.java).
> - Inside the repeat union, there must be a Correlate, or a Join that gets
> implemented as a Correlate due to JoinToCorrelateRule.
> - We have the TransientScan (i.e. the scan on the transient table) on the
> right-hand-side of the correlate/join. Normally, transientScan always appear
> on the LHS of a correlate/join (because when the plan is built they need the
> top of the RelBuilder's stack to get its rowType); however, it can get on the
> RHS due to JoinCommuteRule.
> We can force these conditions and reproduce the issue with the following unit
> test (in EnumerableRepeatUnionTest.java):
> {code:java}
> @Test void testRepeatUnionWithCorrelateWithTransientScanOnItsRight() {
> CalciteAssert.that()
> .with(CalciteConnectionProperty.LEX, Lex.JAVA)
> .with(CalciteConnectionProperty.FORCE_DECORRELATE, false)
> .withSchema("s", new ReflectiveSchema(new HierarchySchema()))
> .query("?")
> .withHook(Hook.PLANNER, (Consumer) planner -> {
> planner.addRule(JoinToCorrelateRule.INSTANCE);
> planner.removeRule(JoinCommuteRule.INSTANCE);
> planner.removeRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
> planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
> })
> .withRel(builder -> {
> builder
> // WITH RECURSIVE delta(empid, name) as (
> // SELECT empid, name FROM emps WHERE empid = 2
> // UNION ALL
> // SELECT e.empid, e.name FROM delta d
> //JOIN hierarchies h ON d.empid =
> h.managerid
> //JOIN emps eON
> h.subordinateid = e.empid
> // )
> // SELECT empid, name FROM delta
> .scan("s", "emps")
> .filter(
> builder.equals(
> builder.field("empid"),
> builder.literal(2)))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .transientScan("#DELTA#");
> RelNode transientScan = builder.build(); // pop the transientScan
> to use it later
> builder
> .scan("s", "hierarchies")
> .push(transientScan) // use the transientScan as right input of
> the join
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "#DELTA#", "empid"),
> builder.field(2, "hierarchies", "managerid")))
> .scan("s", "emps")
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "hierarchies", "subordinateid"),
> builder.field(2, "emps", "empid")))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .repeatUnion("#DELTA#", true);
> return builder.build();
> })
> .explainHookMatches(""
> + "EnumerableRepeatUnion(all=[true])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..4=[{inputs}], expr#5=[2],
> expr#6=[=($t0, $t5)], empid=[$t0], name=[$t2], $condition=[$t6])\n"
> + " EnumerableTableScan(table=[[s, emps]])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + "EnumerableCalc(expr#0..8=[{inputs}], empid=[$t4],
> name=[$t6])\n"
> + " EnumerableCorrelate(correlation=[$cor1],
> joinType=[inner], requiredColumns=[{1}])\n"
> // It is important to have EnumerableCorrelate
[jira] [Updated] (CALCITE-5019) Avoid multiple scans when table is ProjectableFilterableTable
[
https://issues.apache.org/jira/browse/CALCITE-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-5019:
---
Affects Version/s: 1.29.0
(was: 1.30.0)
> Avoid multiple scans when table is ProjectableFilterableTable
> -
>
> Key: CALCITE-5019
> URL: https://issues.apache.org/jira/browse/CALCITE-5019
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.29.0
>Reporter: ChenKai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> {code:sql}
> select a from ProjectableFilterableTable where b = 'xx'
> {code}
> The ProjectableFilterableTable will scan two times when projections and
> filters are different columns.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (CALCITE-5019) Avoid multiple scans when table is ProjectableFilterableTable
[
https://issues.apache.org/jira/browse/CALCITE-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-5019:
---
Fix Version/s: 1.30.0
> Avoid multiple scans when table is ProjectableFilterableTable
> -
>
> Key: CALCITE-5019
> URL: https://issues.apache.org/jira/browse/CALCITE-5019
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.29.0
>Reporter: ChenKai
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.30.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> {code:sql}
> select a from ProjectableFilterableTable where b = 'xx'
> {code}
> The ProjectableFilterableTable will scan two times when projections and
> filters are different columns.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (CALCITE-5019) Avoid multiple scans when table is ProjectableFilterableTable
[
https://issues.apache.org/jira/browse/CALCITE-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-5019.
Resolution: Fixed
Fixed via
https://github.com/apache/calcite/commit/dcbc493bf699d961427952c5efc047b76d859096
Thanks [~514793...@qq.com] for the patch!
> Avoid multiple scans when table is ProjectableFilterableTable
> -
>
> Key: CALCITE-5019
> URL: https://issues.apache.org/jira/browse/CALCITE-5019
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.29.0
>Reporter: ChenKai
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.30.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> {code:sql}
> select a from ProjectableFilterableTable where b = 'xx'
> {code}
> The ProjectableFilterableTable will scan two times when projections and
> filters are different columns.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
