[jira] [Closed] (CALCITE-2948) SqlToRelConverter generates complicated logical plan for subquery
[
https://issues.apache.org/jira/browse/CALCITE-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis closed CALCITE-2948.
Resolution: Duplicate
Hey [~hyuan], I think this is a duplicate of CALCITE-1624. Please, let me know
if that's not the case.
> SqlToRelConverter generates complicated logical plan for subquery
> -
>
> Key: CALCITE-2948
> URL: https://issues.apache.org/jira/browse/CALCITE-2948
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select deptno\n"
> + "from EMP where empno=e.empno+1)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO0=[$9])
> LogicalJoin(condition=[=($0, +($9, 1))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One join would suffice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2944) Lack of test for Aggregate.indicator
[ https://issues.apache.org/jira/browse/CALCITE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800081#comment-16800081 ] Stamatis Zampetakis commented on CALCITE-2944: -- I am not sure if it is worth adding tests for something that is going to be removed. I am not aware of all the use-cases of indicator let's see if [~julianhyde] has an opinion on this. > Lack of test for Aggregate.indicator > > > Key: CALCITE-2944 > URL: https://issues.apache.org/jira/browse/CALCITE-2944 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Attachments: image-2019-03-21-18-19-58-933.png > > > Although Aggregate.indicator is discouraged to use, but no test cases > guarantee indicator can work correctly. I tried to set indicator to always > false in Aggregate constructor, it passed all the tests. > !image-2019-03-21-18-19-58-933.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2658) Introducing more ReduceExpressionRules
[ https://issues.apache.org/jira/browse/CALCITE-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800082#comment-16800082 ] Stamatis Zampetakis commented on CALCITE-2658: -- Hi [~Chunwei Lei], I still didn't find time to have a look on this sorry about that. Since other people already had a look on this it might worth pinging them directly. If nobody responds I will try to find some time to look on this. Thanks for your patience and pushing this forward! > Introducing more ReduceExpressionRules > -- > > Key: CALCITE-2658 > URL: https://issues.apache.org/jira/browse/CALCITE-2658 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > It is useful to have rules reducing Exchange/Sort/SortExchange keys, e.g., > SELECT key,value FROM (SELECT 1 AS key, value FROM src) r DISTRIBUTE BY key; > can be reduced to > SELECT 1 AS key, value FROM src;# Since singleton requirement may already > required by SELECT. > SELECT key,value FROM (SELECT 1 AS key, value FROM src) r ORDER BY key; > can be reduced to > SELECT 1 AS key, value FROM src;# Since ordering on constant is useless. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2947) Type mismatch assertion error when converting NOT IN subquery
[
https://issues.apache.org/jira/browse/CALCITE-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800083#comment-16800083
]
Stamatis Zampetakis commented on CALCITE-2947:
--
Is this related to CALCITE-2901? What do you think [~mgelbana]?
> Type mismatch assertion error when converting NOT IN subquery
> -
>
> Key: CALCITE-2947
> URL: https://issues.apache.org/jira/browse/CALCITE-2947
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryNotIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno not in (select deptno\n"
> + "from EMP where empno=e.empno)";
> sql(sql).ok();
> }
> {code}
> Error:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> INTEGER NOT NULL
> input:
> INTEGER
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:1832)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:140)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:57)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:191)
> at org.apache.calcite.rel.core.Filter.isValid(Filter.java:120)
> at
> org.apache.calcite.rel.logical.LogicalFilter.(LogicalFilter.java:70)
> at
> org.apache.calcite.rel.logical.LogicalFilter.create(LogicalFilter.java:114)
> at
> org.apache.calcite.rel.logical.LogicalFilter.create(LogicalFilter.java:101)
> at
> org.apache.calcite.rel.core.RelFactories$FilterFactoryImpl.createFilter(RelFactories.java:300)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.createJoin(SqlToRelConverter.java:2433)
> {code}
> If we change not in subquery to in subquery, it can run without error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2940) CalcitePrepareImpl always creates a default SqlParser without user's CalciteConnectionConfig
[
https://issues.apache.org/jira/browse/CALCITE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800095#comment-16800095
]
Lai Zhou commented on CALCITE-2940:
---
Duplicate of [ CALCITE-2208|https://issues.apache.org/jira/browse/CALCITE-2208]
. I try to fix this bug, but my PR is quick and dirty, only work for my use
case , so I closed the PR I made.
> CalcitePrepareImpl always creates a default SqlParser without user's
> CalciteConnectionConfig
> --
>
> Key: CALCITE-2940
> URL: https://issues.apache.org/jira/browse/CALCITE-2940
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.18.0
>Reporter: Lai Zhou
>Priority: Major
>
> CalcitePrepareImpl 's method
> {code:java}
> private ParseResult parse_(Context context, String sql, boolean convert,
> boolean analyze, boolean fail)
> {code}
> should use the context to create a SqlParser.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2900) RelStructuredTypeFlattener generates wrong types on nested columns
[ https://issues.apache.org/jira/browse/CALCITE-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated CALCITE-2900: - Summary: RelStructuredTypeFlattener generates wrong types on nested columns (was: RewriteRexShuttle generates wrong type during flatten for structured type when visitInputRef) > RelStructuredTypeFlattener generates wrong types on nested columns > -- > > Key: CALCITE-2900 > URL: https://issues.apache.org/jira/browse/CALCITE-2900 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Will Yu >Priority: Blocker > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > When we leverage Calcite to parse a SQL and generates a RelNode tree on top > of a table with *structured* columns, Calcite throws an error about type > mismatch shown at bottom. > After a bit of debugging, the issue is suspected to happen in > RewriteRexShuttle.visitFieldAccess. When the input is RexInputRef, such as > $36.target_column, the newField type is calculated based on ONLY the index > 36, WITHOUT considering the offset of target_column. The result is new field > type is the type of the first sub-column of column $36, which causes the type > mismatch. > > Error and stacktrace: > java.lang.AssertionError: type mismatch: > ref: > INTEGER NOT NULL > input: > VARCHAR CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary" NOT NULL > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:1832) > at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125) > at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57) > at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112) > at org.apache.calcite.rel.core.Project.isValid(Project.java:186) > at org.apache.calcite.rel.core.Project.(Project.java:83) > at > org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:62) > at > org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:112) > at > org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:100) > at > org.apache.calcite.rel.core.RelFactories$ProjectFactoryImpl.createProject(RelFactories.java:140) > at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1248) > at org.apache.calcite.tools.RelBuilder.projectNamed(RelBuilder.java:1304) > at > org.apache.calcite.sql2rel.RelStructuredTypeFlattener.rewriteRel(RelStructuredTypeFlattener.java:506) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.calcite.util.ReflectUtil.invokeVisitorInternal(ReflectUtil.java:257) > at org.apache.calcite.util.ReflectUtil.invokeVisitor(ReflectUtil.java:214) > at org.apache.calcite.util.ReflectUtil$1.invokeVisitor(ReflectUtil.java:464) > at > org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:759) > at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72) > at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44) > at > org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:754) > at > org.apache.calcite.sql2rel.RelStructuredTypeFlattener.rewrite(RelStructuredTypeFlattener.java:194) > at > org.apache.calcite.sql2rel.SqlToRelConverter.flattenTypes(SqlToRelConverter.java:466) > at org.apache.calcite.prepare.PlannerImpl.rel(PlannerImpl.java:236) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2947) Type mismatch assertion error when converting NOT IN subquery
[
https://issues.apache.org/jira/browse/CALCITE-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800105#comment-16800105
]
Muhammad Gelbana commented on CALCITE-2947:
---
Yes it looks like it. I don't have a stacktrace for the other test case though.
> Type mismatch assertion error when converting NOT IN subquery
> -
>
> Key: CALCITE-2947
> URL: https://issues.apache.org/jira/browse/CALCITE-2947
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryNotIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno not in (select deptno\n"
> + "from EMP where empno=e.empno)";
> sql(sql).ok();
> }
> {code}
> Error:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> INTEGER NOT NULL
> input:
> INTEGER
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:1832)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:140)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:57)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:191)
> at org.apache.calcite.rel.core.Filter.isValid(Filter.java:120)
> at
> org.apache.calcite.rel.logical.LogicalFilter.(LogicalFilter.java:70)
> at
> org.apache.calcite.rel.logical.LogicalFilter.create(LogicalFilter.java:114)
> at
> org.apache.calcite.rel.logical.LogicalFilter.create(LogicalFilter.java:101)
> at
> org.apache.calcite.rel.core.RelFactories$FilterFactoryImpl.createFilter(RelFactories.java:300)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.createJoin(SqlToRelConverter.java:2433)
> {code}
> If we change not in subquery to in subquery, it can run without error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2949) Double aggregate generated for in subquery with distinct
[
https://issues.apache.org/jira/browse/CALCITE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800131#comment-16800131
]
Julian Hyde commented on CALCITE-2949:
--
I strongly believe that this should be fixed in a planner rule (or perhaps
intelligence in RelBuilder), not in SqlToRelConverter. SqlToRelConverter is
necessarily very complex just in order to accomplish its core task; we should
not be making it more complex in order to perform optimizations.
So, someone please work on CALCITE-1172.
> Double aggregate generated for in subquery with distinct
>
>
> Key: CALCITE-2949
> URL: https://issues.apache.org/jira/browse/CALCITE-2949
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select distinct deptno\n"
> + "from EMP where empno=e.empno)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$0], EMPNO=[$1])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One LogicalAggregate would suffice. Although the 2 aggregates can be merged
> by rules described in issue
> [CALCITE-1172|https://issues.apache.org/jira/browse/CALCITE-1172], which is
> not available yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2949) Double aggregate generated for in subquery with distinct
[
https://issues.apache.org/jira/browse/CALCITE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800132#comment-16800132
]
Haisheng Yuan commented on CALCITE-2949:
Agree. I just noticed that AggregateRemoveRule can also do the job for this
pattern. Should I close this issue?
> Double aggregate generated for in subquery with distinct
>
>
> Key: CALCITE-2949
> URL: https://issues.apache.org/jira/browse/CALCITE-2949
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select distinct deptno\n"
> + "from EMP where empno=e.empno)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$0], EMPNO=[$1])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One LogicalAggregate would suffice. Although the 2 aggregates can be merged
> by rules described in issue
> [CALCITE-1172|https://issues.apache.org/jira/browse/CALCITE-1172], which is
> not available yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2949) Double aggregate generated for in subquery with distinct
[
https://issues.apache.org/jira/browse/CALCITE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800133#comment-16800133
]
Julian Hyde commented on CALCITE-2949:
--
Yes, close it.
Perhaps add a `!plan` test case to sub-query.iq.
> Double aggregate generated for in subquery with distinct
>
>
> Key: CALCITE-2949
> URL: https://issues.apache.org/jira/browse/CALCITE-2949
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select distinct deptno\n"
> + "from EMP where empno=e.empno)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$0], EMPNO=[$1])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One LogicalAggregate would suffice. Although the 2 aggregates can be merged
> by rules described in issue
> [CALCITE-1172|https://issues.apache.org/jira/browse/CALCITE-1172], which is
> not available yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (CALCITE-2949) Double aggregate generated for in subquery with distinct
[
https://issues.apache.org/jira/browse/CALCITE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan closed CALCITE-2949.
--
Resolution: Fixed
> Double aggregate generated for in subquery with distinct
>
>
> Key: CALCITE-2949
> URL: https://issues.apache.org/jira/browse/CALCITE-2949
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select distinct deptno\n"
> + "from EMP where empno=e.empno)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$0], EMPNO=[$1])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One LogicalAggregate would suffice. Although the 2 aggregates can be merged
> by rules described in issue
> [CALCITE-1172|https://issues.apache.org/jira/browse/CALCITE-1172], which is
> not available yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2948) SqlToRelConverter generates complicated logical plan for subquery
[
https://issues.apache.org/jira/browse/CALCITE-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800140#comment-16800140
]
Haisheng Yuan commented on CALCITE-2948:
I am not sure. Subquery unnesting is a big topic. These issues are within the
same category, but not necessarily the same case. Because they may go through
different code path.
> SqlToRelConverter generates complicated logical plan for subquery
> -
>
> Key: CALCITE-2948
> URL: https://issues.apache.org/jira/browse/CALCITE-2948
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select deptno\n"
> + "from EMP where empno=e.empno+1)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO0=[$9])
> LogicalJoin(condition=[=($0, +($9, 1))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One join would suffice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2947) Type mismatch assertion error when converting NOT IN subquery
[
https://issues.apache.org/jira/browse/CALCITE-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800154#comment-16800154
]
Haisheng Yuan commented on CALCITE-2947:
Perhaps not the same issue. At least with your proposed changes, the exception
still happens. Needs more scrutiny.
> Type mismatch assertion error when converting NOT IN subquery
> -
>
> Key: CALCITE-2947
> URL: https://issues.apache.org/jira/browse/CALCITE-2947
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryNotIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno not in (select deptno\n"
> + "from EMP where empno=e.empno)";
> sql(sql).ok();
> }
> {code}
> Error:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> INTEGER NOT NULL
> input:
> INTEGER
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:1832)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:140)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:57)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:191)
> at org.apache.calcite.rel.core.Filter.isValid(Filter.java:120)
> at
> org.apache.calcite.rel.logical.LogicalFilter.(LogicalFilter.java:70)
> at
> org.apache.calcite.rel.logical.LogicalFilter.create(LogicalFilter.java:114)
> at
> org.apache.calcite.rel.logical.LogicalFilter.create(LogicalFilter.java:101)
> at
> org.apache.calcite.rel.core.RelFactories$FilterFactoryImpl.createFilter(RelFactories.java:300)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.createJoin(SqlToRelConverter.java:2433)
> {code}
> If we change not in subquery to in subquery, it can run without error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2208) MaterializedViewTable.MATERIALIZATION_CONNECTION breaks lex and case sensitivity for end user
[ https://issues.apache.org/jira/browse/CALCITE-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated CALCITE-2208: - Labels: (was: pull-request-available) > MaterializedViewTable.MATERIALIZATION_CONNECTION breaks lex and case > sensitivity for end user > - > > Key: CALCITE-2208 > URL: https://issues.apache.org/jira/browse/CALCITE-2208 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.15.0, 1.16.0 >Reporter: Piotr Bojko >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > MaterializedViewTable.MATERIALIZATION_CONNECTION used for validating views > uses ORACLE lex by default. Calcite expands the view sql to uppercase so when > schemas used in such view sql are used are declared in lowercase - Calcite > does not find needed objects to resolve and validate the view sql. > It does really not work even when end user creates connection with > lex=oracle, but uses uppercase for the names of its tables. > It would be best when MaterializedViewTable.MATERIALIZATION_CONNECTION would > be replaced by connection of an end user or dynamically created connection > with passed lex from end user connection. > Quick and dirty solution is to create > MaterializedViewTable.MATERIALIZATION_CONNECTION with caseSensitive=false; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2208) MaterializedViewTable.MATERIALIZATION_CONNECTION breaks lex and case sensitivity for end user
[ https://issues.apache.org/jira/browse/CALCITE-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800186#comment-16800186 ] Julian Hyde commented on CALCITE-2208: -- The author of [https://github.com/apache/calcite/pull/1123] has closed the PR. Also, as I commented, I don't think [https://github.com/apache/calcite/pull/647] is suitable. Therefore I am removing the 'pull-request-available' label. > MaterializedViewTable.MATERIALIZATION_CONNECTION breaks lex and case > sensitivity for end user > - > > Key: CALCITE-2208 > URL: https://issues.apache.org/jira/browse/CALCITE-2208 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.15.0, 1.16.0 >Reporter: Piotr Bojko >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > MaterializedViewTable.MATERIALIZATION_CONNECTION used for validating views > uses ORACLE lex by default. Calcite expands the view sql to uppercase so when > schemas used in such view sql are used are declared in lowercase - Calcite > does not find needed objects to resolve and validate the view sql. > It does really not work even when end user creates connection with > lex=oracle, but uses uppercase for the names of its tables. > It would be best when MaterializedViewTable.MATERIALIZATION_CONNECTION would > be replaced by connection of an end user or dynamically created connection > with passed lex from end user connection. > Quick and dirty solution is to create > MaterializedViewTable.MATERIALIZATION_CONNECTION with caseSensitive=false; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2208) MaterializedViewTable.MATERIALIZATION_CONNECTION breaks lex and case sensitivity for end user
[ https://issues.apache.org/jira/browse/CALCITE-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800186#comment-16800186 ] Julian Hyde edited comment on CALCITE-2208 at 3/24/19 10:21 PM: The author of [https://github.com/apache/calcite/pull/1123] has closed the PR. Also, as I commented, I don't think [https://github.com/apache/calcite/pull/647] is suitable. Therefore I am removing the 'pull-request-available' label. Contributors, feel free to revise those pull requests, or come up with new pull requests, and set the 'pull-request-available' label again. If you'd like to discuss what would be a good fix, let's discuss here. was (Author: julianhyde): The author of [https://github.com/apache/calcite/pull/1123] has closed the PR. Also, as I commented, I don't think [https://github.com/apache/calcite/pull/647] is suitable. Therefore I am removing the 'pull-request-available' label. > MaterializedViewTable.MATERIALIZATION_CONNECTION breaks lex and case > sensitivity for end user > - > > Key: CALCITE-2208 > URL: https://issues.apache.org/jira/browse/CALCITE-2208 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.15.0, 1.16.0 >Reporter: Piotr Bojko >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > MaterializedViewTable.MATERIALIZATION_CONNECTION used for validating views > uses ORACLE lex by default. Calcite expands the view sql to uppercase so when > schemas used in such view sql are used are declared in lowercase - Calcite > does not find needed objects to resolve and validate the view sql. > It does really not work even when end user creates connection with > lex=oracle, but uses uppercase for the names of its tables. > It would be best when MaterializedViewTable.MATERIALIZATION_CONNECTION would > be replaced by connection of an end user or dynamically created connection > with passed lex from end user connection. > Quick and dirty solution is to create > MaterializedViewTable.MATERIALIZATION_CONNECTION with caseSensitive=false; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2944) Lack of test for Aggregate.indicator
[
https://issues.apache.org/jira/browse/CALCITE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800187#comment-16800187
]
Julian Hyde commented on CALCITE-2944:
--
Yes, let's remove it (it's been 2 years since we discouraged it, in
CALCITE-1069). I think we should keep the parameter in public API calls, but
throw {{UnsupportedOperationException}} or {{IllegalArgumentException}} if
anyone provides the value {{true}}. Also remove the field where possible.
As noted in CALCITE-1069, you can still use the {{GROUPING}} function to get
what you need.
> Lack of test for Aggregate.indicator
>
>
> Key: CALCITE-2944
> URL: https://issues.apache.org/jira/browse/CALCITE-2944
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Attachments: image-2019-03-21-18-19-58-933.png
>
>
> Although Aggregate.indicator is discouraged to use, but no test cases
> guarantee indicator can work correctly. I tried to set indicator to always
> false in Aggregate constructor, it passed all the tests.
> !image-2019-03-21-18-19-58-933.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2944) Lack of test for Aggregate.indicator
[ https://issues.apache.org/jira/browse/CALCITE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800190#comment-16800190 ] Haisheng Yuan commented on CALCITE-2944: Cool, nice to see it going. It has been used by us without noticing it has been deprecated for 2 years. > Lack of test for Aggregate.indicator > > > Key: CALCITE-2944 > URL: https://issues.apache.org/jira/browse/CALCITE-2944 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Attachments: image-2019-03-21-18-19-58-933.png > > > Although Aggregate.indicator is discouraged to use, but no test cases > guarantee indicator can work correctly. I tried to set indicator to always > false in Aggregate constructor, it passed all the tests. > !image-2019-03-21-18-19-58-933.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2948) SqlToRelConverter generates complicated logical plan for subquery
[
https://issues.apache.org/jira/browse/CALCITE-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800191#comment-16800191
]
Julian Hyde commented on CALCITE-2948:
--
I don't think it is a duplicate. NOT IN sub-queries are inherently more
complicated than IN sub-queries (due to 3-valued boolean logic). There are IN
queries that can be done in 1 join where the NOT IN query requires 2 or more
joins.
So, I think you should re-open.
However, you should test whether this problem still exists if you set
decorrelation = false in {{SqlToRelConverter}}. Things are better if you
decorrelate after sql-to-rel (see {{Tester.isLateDecorrelate()}}) and also use
{{SubQueryRemoveRule}}.
> SqlToRelConverter generates complicated logical plan for subquery
> -
>
> Key: CALCITE-2948
> URL: https://issues.apache.org/jira/browse/CALCITE-2948
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select deptno\n"
> + "from EMP where empno=e.empno+1)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO0=[$9])
> LogicalJoin(condition=[=($0, +($9, 1))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One join would suffice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2712) Add rule to remove null-generating side of a Join
[
https://issues.apache.org/jira/browse/CALCITE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800193#comment-16800193
]
Julian Hyde commented on CALCITE-2712:
--
[~Chunwei Lei], Sure!
> Add rule to remove null-generating side of a Join
> -
>
> Key: CALCITE-2712
> URL: https://issues.apache.org/jira/browse/CALCITE-2712
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Add a rule to remove the null-generating side of a Join. Here is an example
> where we eliminate the "many" side of a "one-to-many" join:
> {code:sql}
> # Example 1: one-to-many
> SELECT c.id, COUNT(DISTINCT o.productId)
> FROM Customers AS c
> LEFT JOIN SupportCalls AS s ON c.id = s.customerId
> LEFT JOIN Orders AS o ON c.id = o.customerId{code}
> We can remove {{SupportCalls}} and the join to it, so the query becomes
> {code:sql}
> SELECT c.id, COUNT(DISTINCT o.productId)
> FROM Customers AS c
> LEFT JOIN Orders AS o ON c.id = o.customerId{code}
> Necessary conditions are:
> # no columns from {{SupportCalls}} are used
> # the join is LEFT, so customers will not be eliminated if they have no
> support calls,
> # there is an Aggregate on top, so we don't care if there are >1 support call
> per customer.
> A simpler example of one-to-many:
> {code:sql}
> # Example 2: simpler one-to-many
> SELECT DISTINCT c.id
> FROM Customers AS c
> LEFT JOIN SupportCalls AS s ON c.id = s.customerId{code}
> An example of many-to-one, where we eliminate the "one" side ({{Orders}}):
> {code:sql}
> # Example 3: many-to-one
> SELECT c.id, p.color
> FROM LineItems AS i
> LEFT JOIN Orders AS o ON o.id = i.orderId
> LEFT JOIN Products AS p ON p.id = i.orderId{code}
> so that the query becomes
> {code:sql}
> SELECT c.id, p.color
> FROM LineItems AS i
> LEFT JOIN Products AS p ON p.id = i.orderId{code}
> Here, necessary side-conditions are:
> # no columns from {{Orders}} are used;
> # unique key on {{Orders.id}}.
> We do not require aggregation, because the primary key on {{Orders.id}}
> ensures that {{Orders}} contributes at most one row.
> We can deal with similar cases like
> {code:sql}
> # Example 4: many-to-one, column aliasing required
> SELECT c.id, p.color
> FROM LineItems AS i
> LEFT JOIN Orders AS o ON o.id = i.orderId
> LEFT JOIN Products AS p ON p.id = o.id{code}
> if we use aliasing ({{o.id}} = {{i.orderId}}) and a foreign key that ensures
> the existence of an record in {{Orders}}.
> For examples 1 and 2 (one-to-many), we would need to match {{Aggregate}} on
> {{Join}}, therefore make a variant of {{AggregateJoinTransposeRule}}.
> For examples 3 and 4 (many-to-one or one-to-one)), we would match {{Project}}
> on {{Join}}, therefore make a variant of {{ProjectJoinTransposeRule}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2948) SqlToRelConverter generates complicated logical plan for subquery
[
https://issues.apache.org/jira/browse/CALCITE-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800209#comment-16800209
]
Haisheng Yuan commented on CALCITE-2948:
It looks like the problem still exists.
{code:java}
private Sql checkSubQuery(String sql) {
final HepProgram program = new HepProgramBuilder()
.addRuleInstance(SubQueryRemoveRule.PROJECT)
.addRuleInstance(SubQueryRemoveRule.FILTER)
.addRuleInstance(SubQueryRemoveRule.JOIN)
.build();
return sql(sql).with(new HepPlanner(program)).expand(false);
}
@Test public void testWhereInCorrelated() {
final String sql = "select deptno\n"
+ "from EMP e\n"
+ "where deptno in (select deptno\n"
+ "from EMP where empno=e.empno+1)";
checkSubQuery(sql).withDecorrelation(false).withLateDecorrelation(true).check();
}
{code}
Plan before:
{code:java}
LogicalProject(DEPTNO=[$7])
LogicalFilter(condition=[IN($7, {
LogicalProject(DEPTNO=[$7])
LogicalFilter(condition=[=($0, +($cor0.EMPNO, 1))])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
})], variablesSet=[[$cor0]])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
Plan Mid:
{code:java}
LogicalProject(DEPTNO=[$7])
LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
LogicalFilter(condition=[=($7, $9)])
LogicalCorrelate(correlation=[$cor0], joinType=[inner],
requiredColumns=[{0}])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalFilter(condition=[=($0, +($cor0.EMPNO, 1))])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
Plan After:
{code:java}
LogicalProject(DEPTNO=[$7])
LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalAggregate(group=[{0, 1}])
LogicalProject(DEPTNO=[$7], EMPNO0=[$9])
LogicalJoin(condition=[=($0, +($9, 1))], joinType=[inner])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
LogicalProject(EMPNO=[$0])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
> SqlToRelConverter generates complicated logical plan for subquery
> -
>
> Key: CALCITE-2948
> URL: https://issues.apache.org/jira/browse/CALCITE-2948
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select deptno\n"
> + "from EMP where empno=e.empno+1)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO0=[$9])
> LogicalJoin(condition=[=($0, +($9, 1))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One join would suffice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (CALCITE-2948) SqlToRelConverter generates complicated logical plan for subquery
[
https://issues.apache.org/jira/browse/CALCITE-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan reopened CALCITE-2948:
> SqlToRelConverter generates complicated logical plan for subquery
> -
>
> Key: CALCITE-2948
> URL: https://issues.apache.org/jira/browse/CALCITE-2948
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select deptno\n"
> + "from EMP where empno=e.empno+1)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO0=[$9])
> LogicalJoin(condition=[=($0, +($9, 1))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One join would suffice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (CALCITE-2790) AggregateJoinTransposeRule incorrectly pushes down distinct count into join
[ https://issues.apache.org/jira/browse/CALCITE-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Risden closed CALCITE-2790. - > AggregateJoinTransposeRule incorrectly pushes down distinct count into join > --- > > Key: CALCITE-2790 > URL: https://issues.apache.org/jira/browse/CALCITE-2790 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Andrew Tsvelodub >Assignee: Julian Hyde >Priority: Major > Fix For: 1.19.0 > > > AggregateJoinTransposeRule.EXTENDED can push down aggregate functions into > the join. > For Distinct Count (and other distinct aggregate functions) this should not > be performed the same way as for simple count, since distinct count is not a > decomposable function. [1] > While this can be done in theory, by adding the distinct column to the list > of groups in below aggregate[2], I want to fix the buggy behavior first, > before trying to improve the existing rule. > [1] Dynamic Programming: The Next Step, M. Eich, G. Moerkotte > ([https://ub-madoc.bib.uni-mannheim.de/37228/1/main.pdf]) > [2] Including Group-By in Query Optimization, S. Chaudhuri, K. Shim > ([https://pdfs.semanticscholar.org/3079/5447cec18753254edbbd7839f0afa58b2a39.pdf]) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CALCITE-2712) Add rule to remove null-generating side of a Join
[
https://issues.apache.org/jira/browse/CALCITE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei reassigned CALCITE-2712:
Assignee: Chunwei Lei
> Add rule to remove null-generating side of a Join
> -
>
> Key: CALCITE-2712
> URL: https://issues.apache.org/jira/browse/CALCITE-2712
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Chunwei Lei
>Priority: Major
>
> Add a rule to remove the null-generating side of a Join. Here is an example
> where we eliminate the "many" side of a "one-to-many" join:
> {code:sql}
> # Example 1: one-to-many
> SELECT c.id, COUNT(DISTINCT o.productId)
> FROM Customers AS c
> LEFT JOIN SupportCalls AS s ON c.id = s.customerId
> LEFT JOIN Orders AS o ON c.id = o.customerId{code}
> We can remove {{SupportCalls}} and the join to it, so the query becomes
> {code:sql}
> SELECT c.id, COUNT(DISTINCT o.productId)
> FROM Customers AS c
> LEFT JOIN Orders AS o ON c.id = o.customerId{code}
> Necessary conditions are:
> # no columns from {{SupportCalls}} are used
> # the join is LEFT, so customers will not be eliminated if they have no
> support calls,
> # there is an Aggregate on top, so we don't care if there are >1 support call
> per customer.
> A simpler example of one-to-many:
> {code:sql}
> # Example 2: simpler one-to-many
> SELECT DISTINCT c.id
> FROM Customers AS c
> LEFT JOIN SupportCalls AS s ON c.id = s.customerId{code}
> An example of many-to-one, where we eliminate the "one" side ({{Orders}}):
> {code:sql}
> # Example 3: many-to-one
> SELECT c.id, p.color
> FROM LineItems AS i
> LEFT JOIN Orders AS o ON o.id = i.orderId
> LEFT JOIN Products AS p ON p.id = i.orderId{code}
> so that the query becomes
> {code:sql}
> SELECT c.id, p.color
> FROM LineItems AS i
> LEFT JOIN Products AS p ON p.id = i.orderId{code}
> Here, necessary side-conditions are:
> # no columns from {{Orders}} are used;
> # unique key on {{Orders.id}}.
> We do not require aggregation, because the primary key on {{Orders.id}}
> ensures that {{Orders}} contributes at most one row.
> We can deal with similar cases like
> {code:sql}
> # Example 4: many-to-one, column aliasing required
> SELECT c.id, p.color
> FROM LineItems AS i
> LEFT JOIN Orders AS o ON o.id = i.orderId
> LEFT JOIN Products AS p ON p.id = o.id{code}
> if we use aliasing ({{o.id}} = {{i.orderId}}) and a foreign key that ensures
> the existence of an record in {{Orders}}.
> For examples 1 and 2 (one-to-many), we would need to match {{Aggregate}} on
> {{Join}}, therefore make a variant of {{AggregateJoinTransposeRule}}.
> For examples 3 and 4 (many-to-one or one-to-one)), we would match {{Project}}
> on {{Join}}, therefore make a variant of {{ProjectJoinTransposeRule}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2950) Avatica DriverVersion.load leaks InputStream
Xiaobo Liao created CALCITE-2950:
Summary: Avatica DriverVersion.load leaks InputStream
Key: CALCITE-2950
URL: https://issues.apache.org/jira/browse/CALCITE-2950
Project: Calcite
Issue Type: Bug
Components: avatica
Affects Versions: avatica-1.13.0
Environment: CentOS release 6.9 (Final)
Reporter: Xiaobo Liao
Fix For: next
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed. Though it is nonsense
to calling DriverVersion.load repeatedly, but the leak is there.
{{}}
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true) {
DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties",
"Kylin JDBC Driver", "unknown version", "Kylin", "unknown version");
System.out.println(version);
Thread.sleep(100l);
}
}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoader().getResourceAsStream(resourceName);
{code}
The code to reproduce the leak is at
https://github.com/leonliao/calcite-avatica-driver-version-test. Follow the
README to reproduce the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (CALCITE-2900) RelStructuredTypeFlattener generates wrong types on nested columns
[ https://issues.apache.org/jira/browse/CALCITE-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Yu reassigned CALCITE-2900: Assignee: Will Yu > RelStructuredTypeFlattener generates wrong types on nested columns > -- > > Key: CALCITE-2900 > URL: https://issues.apache.org/jira/browse/CALCITE-2900 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Will Yu >Assignee: Will Yu >Priority: Blocker > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > When we leverage Calcite to parse a SQL and generates a RelNode tree on top > of a table with *structured* columns, Calcite throws an error about type > mismatch shown at bottom. > After a bit of debugging, the issue is suspected to happen in > RewriteRexShuttle.visitFieldAccess. When the input is RexInputRef, such as > $36.target_column, the newField type is calculated based on ONLY the index > 36, WITHOUT considering the offset of target_column. The result is new field > type is the type of the first sub-column of column $36, which causes the type > mismatch. > > Error and stacktrace: > java.lang.AssertionError: type mismatch: > ref: > INTEGER NOT NULL > input: > VARCHAR CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary" NOT NULL > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:1832) > at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125) > at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57) > at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112) > at org.apache.calcite.rel.core.Project.isValid(Project.java:186) > at org.apache.calcite.rel.core.Project.(Project.java:83) > at > org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:62) > at > org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:112) > at > org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:100) > at > org.apache.calcite.rel.core.RelFactories$ProjectFactoryImpl.createProject(RelFactories.java:140) > at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1248) > at org.apache.calcite.tools.RelBuilder.projectNamed(RelBuilder.java:1304) > at > org.apache.calcite.sql2rel.RelStructuredTypeFlattener.rewriteRel(RelStructuredTypeFlattener.java:506) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.calcite.util.ReflectUtil.invokeVisitorInternal(ReflectUtil.java:257) > at org.apache.calcite.util.ReflectUtil.invokeVisitor(ReflectUtil.java:214) > at org.apache.calcite.util.ReflectUtil$1.invokeVisitor(ReflectUtil.java:464) > at > org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:759) > at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72) > at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44) > at > org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:754) > at > org.apache.calcite.sql2rel.RelStructuredTypeFlattener.rewrite(RelStructuredTypeFlattener.java:194) > at > org.apache.calcite.sql2rel.SqlToRelConverter.flattenTypes(SqlToRelConverter.java:466) > at org.apache.calcite.prepare.PlannerImpl.rel(PlannerImpl.java:236) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-2950) Avatica DriverVersion.load leaks InputStream
[
https://issues.apache.org/jira/browse/CALCITE-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-2950:
Labels: pull-request-available (was: )
> Avatica DriverVersion.load leaks InputStream
>
>
> Key: CALCITE-2950
> URL: https://issues.apache.org/jira/browse/CALCITE-2950
> Project: Calcite
> Issue Type: Bug
> Components: avatica
>Affects Versions: avatica-1.13.0
> Environment: CentOS release 6.9 (Final)
>Reporter: Xiaobo Liao
>Priority: Minor
> Labels: pull-request-available
> Fix For: next
>
>
> Kylin's JDBC
> Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
> is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
> DriverVersion.load, native memory leak can be observed. Though it is nonsense
> to calling DriverVersion.load repeatedly, but the leak is there.
> {{}}
> package org.apache.calcite.avatica.test;
> import org.apache.calcite.avatica.DriverVersion;
> import org.apache.calcite.avatica.remote.Driver;
>
> /**
> * This class demonstrates the native memory leak caused by DriverVersion.load,
> * which has not closed the opened InputStream.
> */
> public class DriverVersionTest {
>
> public static void main(String[] args) throws Exception {
> while (true) {
> DriverVersion version = DriverVersion.load(Driver.class,
> "org-apache-kylin-jdbc.properties",
> "Kylin JDBC Driver", "unknown version", "Kylin", "unknown version");
> System.out.println(version);
> Thread.sleep(100l);
> }
> }
>
> By using jemalloc's jeprof utility, the leak is caused by below stack trace
> at java.util.zip.Inflater.inflate()
> at java.util.zip.InflaterInputStream.read()
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.FilterInputStream.read(FilterInputStream.java:107)
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:353)
> at java.util.Properties.load(Properties.java:341)
> at
> [org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
> at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
> at
> [org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
> at org.apache.kylin.jdbc.Driver.(Driver.java:70)
>
> By inspecting the code, the InputStream was not closed after use.
>
> {code:java}
> final InputStream inStream =
> driverClass.getClassLoader().getResourceAsStream(resourceName);
> {code}
>
> The code to reproduce the leak is at
> https://github.com/leonliao/calcite-avatica-driver-version-test. Follow the
> README to reproduce the problem.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2950) Avatica DriverVersion.load leaks InputStream
[
https://issues.apache.org/jira/browse/CALCITE-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobo Liao updated CALCITE-2950:
-
Description:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed. Though it is nonsense
to calling DriverVersion.load repeatedly, but the leak is there.
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true)
{ DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown version",
"Kylin", "unknown version"); System.out.println(version);
Thread.sleep(100l); }
}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoader().getResourceAsStream(resourceName);
{code}
The code to reproduce the leak is at
[https://github.com/leonliao/calcite-avatica-driver-version-test]. Follow the
README to reproduce the problem.
was:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed. Though it is nonsense
to calling DriverVersion.load repeatedly, but the leak is there.
{{}}
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true) {
DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties",
"Kylin JDBC Driver", "unknown version", "Kylin", "unknown version");
System.out.println(version);
Thread.sleep(100l);
}
}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoader().getResourceAsStream(resourceName);
{code}
The code to reproduce the leak is at
https://github.com/leonliao/calcite-avatica-
[jira] [Commented] (CALCITE-2936) Simplify EXISTS or NOT EXISTS sub-query that has "GROUP BY ()"
[
https://issues.apache.org/jira/browse/CALCITE-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800380#comment-16800380
]
Danny Chan commented on CALCITE-2936:
-
So the MinRowCount and MaxRowCount are all estimated but safe bounds. We should
update the java doc i think.
> Simplify EXISTS or NOT EXISTS sub-query that has "GROUP BY ()"
> --
>
> Key: CALCITE-2936
> URL: https://issues.apache.org/jira/browse/CALCITE-2936
> Project: Calcite
> Issue Type: New Feature
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> An EXISTS or NOT EXISTS sub-query whose inner child is an aggregate with no
> grouping columns should be simplified to a Boolean constant.
> Example:
> {code:java}
> exists(select sum(i) from X) --> true
> not exists(select sum(i) from X) --> false
> {code}
> Repro:
> {code:java}
> @Test public void testExistentialSubquery() {
> final String sql = "SELECT e1.empno\n"
> + "FROM emp e1 where exists\n"
> + "(select avg(sal) from emp e2 where e1.empno = e2.empno )";
> sql(sql).decorrelate(true).ok();
> }
> {code}
> We got plan:
> {code:java}
> LogicalProject(EMPNO=[$0])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8], EMPNO0=[CAST($9):INTEGER],
> $f1=[CAST($10):BOOLEAN])
> LogicalJoin(condition=[=($0, $9)], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0}], agg#0=[MIN($1)])
> LogicalProject(EMPNO=[$0], $f0=[true])
> LogicalAggregate(group=[{0}], EXPR$0=[AVG($1)])
> LogicalProject(EMPNO=[$0], SAL=[$5])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> The preferred plan should be:
> {code:java}
> LogicalProject(EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2950) Avatica DriverVersion.load leaks InputStream
[
https://issues.apache.org/jira/browse/CALCITE-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobo Liao updated CALCITE-2950:
-
Attachment: stream_not_closed.png
> Avatica DriverVersion.load leaks InputStream
>
>
> Key: CALCITE-2950
> URL: https://issues.apache.org/jira/browse/CALCITE-2950
> Project: Calcite
> Issue Type: Bug
> Components: avatica
>Affects Versions: avatica-1.13.0
> Environment: CentOS release 6.9 (Final)
>Reporter: Xiaobo Liao
>Priority: Minor
> Labels: pull-request-available
> Fix For: next
>
> Attachments: stream_not_closed.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Kylin's JDBC
> Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
> is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
> DriverVersion.load, native memory leak can be observed. Though it is nonsense
> to calling DriverVersion.load repeatedly, but the leak is there.
> package org.apache.calcite.avatica.test;
> import org.apache.calcite.avatica.DriverVersion;
> import org.apache.calcite.avatica.remote.Driver;
>
> /**
> * This class demonstrates the native memory leak caused by
> DriverVersion.load,
> * which has not closed the opened InputStream.
> */
> public class DriverVersionTest {
>
> public static void main(String[] args) throws Exception {
> while (true)
> { DriverVersion version = DriverVersion.load(Driver.class,
> "org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown
> version", "Kylin", "unknown version"); System.out.println(version);
> Thread.sleep(100l); }
>
> }
>
> By using jemalloc's jeprof utility, the leak is caused by below stack trace
> at java.util.zip.Inflater.inflate()
> at java.util.zip.InflaterInputStream.read()
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.FilterInputStream.read(FilterInputStream.java:107)
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:353)
> at java.util.Properties.load(Properties.java:341)
> at
> [org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
> at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
> at
> [org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
> at org.apache.kylin.jdbc.Driver.(Driver.java:70)
>
> By inspecting the code, the InputStream was not closed after use.
>
> {code:java}
> final InputStream inStream =
> driverClass.getClassLoader().getResourceAsStream(resourceName);
> {code}
>
> The code to reproduce the leak is at
> [https://github.com/leonliao/calcite-avatica-driver-version-test]. Follow the
> README to reproduce the problem.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2950) Avatica DriverVersion.load leaks InputStream
[
https://issues.apache.org/jira/browse/CALCITE-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobo Liao updated CALCITE-2950:
-
Description:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed, see attached memory
profile diagram(from jeprof). Though it is nonsense to calling
DriverVersion.load repeatedly, but the leak is there.
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true)
{ DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown version",
"Kylin", "unknown version"); System.out.println(version);
Thread.sleep(100l); }
}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoader().getResourceAsStream(resourceName);
{code}
The code to reproduce the leak is at
[https://github.com/leonliao/calcite-avatica-driver-version-test]. Follow the
README to reproduce the problem.
was:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed. Though it is nonsense
to calling DriverVersion.load repeatedly, but the leak is there.
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true)
{ DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown version",
"Kylin", "unknown version"); System.out.println(version);
Thread.sleep(100l); }
}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoader().getResourceAsStream(resourceName);
{code}
The code to reproduce th
[jira] [Commented] (CALCITE-883) Support RESPECT NULLS, IGNORE NULLS option for LEAD, LAG, FIRST_VALUE, LAST_VALUE window functions
[ https://issues.apache.org/jira/browse/CALCITE-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800381#comment-16800381 ] Hongze Zhang commented on CALCITE-883: -- It would be great if anyone else could share some thoughts on this topic. And as you can see from the table above, there is already a field "functionQuantifier"[1] in class SqlBasicCall. At this time I would suspect that the field's location is also discussable -- as a "call" is to an operator but not necessarily a function, do we really have to bind such a property to "call"s? (Sorry may be a little out of our topic) [1] https://github.com/apache/calcite/blob/a648f9c12309cc253628930b0cab98591caa66ab/core/src/main/java/org/apache/calcite/sql/SqlBasicCall.java#L31 > Support RESPECT NULLS, IGNORE NULLS option for LEAD, LAG, FIRST_VALUE, > LAST_VALUE window functions > -- > > Key: CALCITE-883 > URL: https://issues.apache.org/jira/browse/CALCITE-883 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Sean Hsuan-Yi Chu >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > This issue was brought up in Drill community: > (https://issues.apache.org/jira/browse/DRILL-3597) > For convenience, I copied and pasted the reporter Khurram Faraaz's > description here: > "The SQL standard defines a RESPECT NULLS or IGNORE NULLS option for lead, > lag, first_value, and last_value window functions." > In Calcite, we should have been able to recognized this option. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-2950) Avatica DriverVersion.load leaks InputStream
[
https://issues.apache.org/jira/browse/CALCITE-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobo Liao updated CALCITE-2950:
-
Description:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed, see attached memory
profile diagram(from jeprof). Though it is nonsense to calling
DriverVersion.load repeatedly, but the leak is there.
{{
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true)
{ DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown version",
"Kylin", "unknown version"); System.out.println(version);
Thread.sleep(100l); }
}
}}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoader().getResourceAsStream(resourceName);
{code}
The code to reproduce the leak is at
[https://github.com/leonliao/calcite-avatica-driver-version-test]. Follow the
README to reproduce the problem.
was:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed, see attached memory
profile diagram(from jeprof). Though it is nonsense to calling
DriverVersion.load repeatedly, but the leak is there.
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true)
{ DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown version",
"Kylin", "unknown version"); System.out.println(version);
Thread.sleep(100l); }
}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoader().getResourceA
[jira] [Updated] (CALCITE-2950) Avatica DriverVersion.load leaks InputStream
[
https://issues.apache.org/jira/browse/CALCITE-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobo Liao updated CALCITE-2950:
-
Description:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed, see attached memory
profile diagram(from jeprof). Though it is nonsense to calling
DriverVersion.load repeatedly, but the leak is there.
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true) {
DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown version",
"Kylin", "unknown version"); System.out.println(version);
Thread.sleep(100l);
}
}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoader().getResourceAsStream(resourceName);
{code}
The code to reproduce the leak is at
[https://github.com/leonliao/calcite-avatica-driver-version-test]. Follow the
README to reproduce the problem.
was:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed, see attached memory
profile diagram(from jeprof). Though it is nonsense to calling
DriverVersion.load repeatedly, but the leak is there.
{{
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true)
{ DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown version",
"Kylin", "unknown version"); System.out.println(version);
Thread.sleep(100l); }
}
}}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoade
[jira] [Updated] (CALCITE-2950) Avatica DriverVersion.load leaks InputStream
[
https://issues.apache.org/jira/browse/CALCITE-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobo Liao updated CALCITE-2950:
-
Description:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invoking
{code:java}
//code from Application code.
Driver driver = (Driver)
Class.forName("org.apache.kylin.jdbc.Driver").newInstance();
{code}
, which repeatedly invokes
{code:java}
//code from Kylin JDBC
DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown version",
"Kylin", "unknown version");
{code}
Then the native memory leak can be observed, see attached memory profile
diagram(from jeprof).
And the leaking stack trace is below.
at java.util.zip.Inflater.inflate
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the Avatica code, the InputStream was not closed after use.
{code:java}
//code from Avatica DriverVersion.load
final InputStream inStream =
driverClass.getClassLoader().getResourceAsStream(resourceName);
{code}
Though it is nonsense to calling DriverVersion.load repeatedly, but the leak
is there.
The code to reproduce the leak is at
[https://github.com/leonliao/calcite-avatica-driver-version-test]. Follow the
README to reproduce the problem.
was:
Kylin's JDBC
Driver([https://github.com/apache/kylin/tree/master/jdbc)|https://github.com/apache/kylin/tree/master/jdbc),]
is based on Avatica. While using Kylin JDBC Driver, if repeatedly invokes
DriverVersion.load, native memory leak can be observed, see attached memory
profile diagram(from jeprof). Though it is nonsense to calling
DriverVersion.load repeatedly, but the leak is there.
package org.apache.calcite.avatica.test;
import org.apache.calcite.avatica.DriverVersion;
import org.apache.calcite.avatica.remote.Driver;
/**
* This class demonstrates the native memory leak caused by DriverVersion.load,
* which has not closed the opened InputStream.
*/
public class DriverVersionTest {
public static void main(String[] args) throws Exception {
while (true) {
DriverVersion version = DriverVersion.load(Driver.class,
"org-apache-kylin-jdbc.properties", "Kylin JDBC Driver", "unknown version",
"Kylin", "unknown version"); System.out.println(version);
Thread.sleep(100l);
}
}
By using jemalloc's jeprof utility, the leak is caused by below stack trace
at java.util.zip.Inflater.inflate()
at java.util.zip.InflaterInputStream.read()
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.DriverVersion.load(DriverVersion.java:104|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.driverversion.load%28driverversion.java:104/])
at org.apache.kylin.jdbc.Driver.createDriverVersion(Driver.java:88)
at
[org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.UnregisteredDriver|http://org.apache.kylin.jdbc.shaded.org.apache.calcite.avatica.unregistereddriver/].(UnregisteredDriver.java:56)
at org.apache.kylin.jdbc.Driver.(Driver.java:70)
By inspecting the code, the InputStream was not closed after use.
{code:java}
final InputStream inStream =
driverClass.getClassLoader().getResourceAsStream(resourceName);
{code}
The code to reproduce the leak is at
[https://github.com/leonliao/calcite-avatica-driver-version-test]. Follow the
README to reproduce the problem.
> Avatica DriverVersion.load leaks InputStream
>
>
>
[jira] [Comment Edited] (CALCITE-883) Support RESPECT NULLS, IGNORE NULLS option for LEAD, LAG, FIRST_VALUE, LAST_VALUE window functions
[ https://issues.apache.org/jira/browse/CALCITE-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800399#comment-16800399 ] Chunwei Lei edited comment on CALCITE-883 at 3/25/19 4:25 AM: -- [~zhztheplayer] As far as I see, I am afraid that your proposal may can not work(Maybe I am wrong). Would you mind sharing some code? It may help us to agree upon which way is best. Thanks. was (Author: chunwei lei): As far as I see, I am afraid that your proposal may can not work. Would you mind sharing some code? It may help us to agree upon which way is best. Thanks. > Support RESPECT NULLS, IGNORE NULLS option for LEAD, LAG, FIRST_VALUE, > LAST_VALUE window functions > -- > > Key: CALCITE-883 > URL: https://issues.apache.org/jira/browse/CALCITE-883 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Sean Hsuan-Yi Chu >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > This issue was brought up in Drill community: > (https://issues.apache.org/jira/browse/DRILL-3597) > For convenience, I copied and pasted the reporter Khurram Faraaz's > description here: > "The SQL standard defines a RESPECT NULLS or IGNORE NULLS option for lead, > lag, first_value, and last_value window functions." > In Calcite, we should have been able to recognized this option. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-883) Support RESPECT NULLS, IGNORE NULLS option for LEAD, LAG, FIRST_VALUE, LAST_VALUE window functions
[ https://issues.apache.org/jira/browse/CALCITE-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800399#comment-16800399 ] Chunwei Lei commented on CALCITE-883: - As far as I see, I am afraid that your proposal may can not work. Would you mind sharing some code? It may help us to agree upon which way is best. Thanks. > Support RESPECT NULLS, IGNORE NULLS option for LEAD, LAG, FIRST_VALUE, > LAST_VALUE window functions > -- > > Key: CALCITE-883 > URL: https://issues.apache.org/jira/browse/CALCITE-883 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Sean Hsuan-Yi Chu >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > This issue was brought up in Drill community: > (https://issues.apache.org/jira/browse/DRILL-3597) > For convenience, I copied and pasted the reporter Khurram Faraaz's > description here: > "The SQL standard defines a RESPECT NULLS or IGNORE NULLS option for lead, > lag, first_value, and last_value window functions." > In Calcite, we should have been able to recognized this option. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
