[jira] [Commented] (CALCITE-4030) Assert error during top-down optimization with Project/Filter Traits passdown and derivation
[
https://issues.apache.org/jira/browse/CALCITE-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119290#comment-17119290
]
Rui Wang commented on CALCITE-4030:
---
This Jira is opened to keep a tracking for this special assert error. We will
need to investigate the root case before make top-down optimization as default.
> Assert error during top-down optimization with Project/Filter Traits passdown
> and derivation
> -
>
> Key: CALCITE-4030
> URL: https://issues.apache.org/jira/browse/CALCITE-4030
> Project: Calcite
> Issue Type: Task
>Reporter: Rui Wang
>Priority: Major
>
> For example, with https://github.com/apache/calcite/pull/1985, and enable
> top-down opt by set "calcite.planner.topdown.opt=true" in saffron.properties,
> run test case SortRemoveRuleTest.removeSortOverEnumerableHashJoin, we can see:
> {code:java}
> java.lang.AssertionError
> at
> org.apache.calcite.plan.volcano.OptimizeTask$RelNodeOptTask.execute(OptimizeTask.java:232)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:553)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:327)
> at
> org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:362)
> at
> org.apache.calcite.rel.rules.SortRemoveRuleTest.transform(SortRemoveRuleTest.java:77)
> at
> org.apache.calcite.rel.rules.SortRemoveRuleTest.removeSortOverEnumerableHashJoin(SortRemoveRuleTest.java:102)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4030) Assert error during top-down optimization with Project/Filter Traits passdown and derivation
Rui Wang created CALCITE-4030:
-
Summary: Assert error during top-down optimization with
Project/Filter Traits passdown and derivation
Key: CALCITE-4030
URL: https://issues.apache.org/jira/browse/CALCITE-4030
Project: Calcite
Issue Type: Task
Reporter: Rui Wang
For example, with https://github.com/apache/calcite/pull/1985, and enable
top-down opt by set "calcite.planner.topdown.opt=true" in saffron.properties,
run test case SortRemoveRuleTest.removeSortOverEnumerableHashJoin, we can see:
{code:java}
java.lang.AssertionError
at
org.apache.calcite.plan.volcano.OptimizeTask$RelNodeOptTask.execute(OptimizeTask.java:232)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:553)
at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:327)
at
org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:362)
at
org.apache.calcite.rel.rules.SortRemoveRuleTest.transform(SortRemoveRuleTest.java:77)
at
org.apache.calcite.rel.rules.SortRemoveRuleTest.removeSortOverEnumerableHashJoin(SortRemoveRuleTest.java:102)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3963) Maintain logical properties at RelSet (equivalent group) instead of RelNode
[ https://issues.apache.org/jira/browse/CALCITE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119284#comment-17119284 ] Xiening Dai commented on CALCITE-3963: -- I create a PR - https://github.com/apache/calcite/pull/1992/files please take a look. It only addresses row count, but I could make similar changes for other logical properties. But just want to hear some feedback before moving forward. The basic idea is to always use set.rel (renamed into set.originalRel) for reporting logical properties of a given set. And we add a new interface LogicalNode which can report the confidence of its stats estimation. Whenever a new LogicalNode is added into the set, we check if its confidence level is higher than set.rel, and update it if that is the case. There are some use cases where the confidence level is useful. For example, we treat TableScan with high confidence level and that would make sure in materialized view, the table scan node would be used for reporting row count instead of other equivalent rel nodes. That makes sense since it's been materialized and we can safely assume that it has the most accurate row count. And we definitely can extend that interface to handle more complex scenarios in the future. > Maintain logical properties at RelSet (equivalent group) instead of RelNode > --- > > Key: CALCITE-3963 > URL: https://issues.apache.org/jira/browse/CALCITE-3963 > Project: Calcite > Issue Type: Bug >Reporter: Xiening Dai >Assignee: Xiening Dai >Priority: Major > > Currently the logical properties (such as row count, distinct row count, etc) > are maintained at RelNode level. This creates a number of meta data > consistency problems, e.g. CALCITE-1048, CALCITE-2166. > In theory, all RelNodes in a RelSet should share the same logical properties > per definition of relational equivalence. So it makes more sense to keep > logical properties at RelSet level, rather than the RelNode. And such > properties shouldn't change when new sub set is created or subset's best is > changed. > Specifically I think below build in metadata should fall into the logical > properties category - > Selectivity > UniqueKeys > ColumnUniqueness > RowCount > MaxRowCount > MinRowCount > DistinctRowCount > Size (averageRowSize, averageColumnSize) > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4019) Visit SqlInsert with SqlShuttle cause NullPointerException
[
https://issues.apache.org/jira/browse/CALCITE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yanlin updated CALCITE-4019:
-
Fix Version/s: 1.24.0
> Visit SqlInsert with SqlShuttle cause NullPointerException
> --
>
> Key: CALCITE-4019
> URL: https://issues.apache.org/jira/browse/CALCITE-4019
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Major
> Fix For: 1.24.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When visit a SqlInsert with SqlShuttle, got NPE, the stack trace is
> {noformat}
> java.lang.NullPointerException
> at com.google.common.collect.Iterators$8.transform(Iterators.java:817)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at org.apache.calcite.sql.parser.SqlParserPos.sum(SqlParserPos.java:274)
> at
> org.apache.calcite.sql.parser.SqlParserPos.plusAll(SqlParserPos.java:183)
> at org.apache.calcite.sql.SqlOperator.createCall(SqlOperator.java:251)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.result(SqlShuttle.java:117)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.result(SqlShuttle.java:101)
> at org.apache.calcite.sql.util.SqlShuttle.visit(SqlShuttle.java:67)
> at org.apache.calcite.sql.util.SqlShuttle.visit(SqlShuttle.java:39)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
> at
> org.apache.calcite.sql.parser.SqlParserTest.testSqlParserPosPlus(SqlParserTest.java:8791)
> {noformat}
> Add this case in SqlParserTest to reproduce.
> return a new same SqlIdentifier for each SqlIdentifier, make sure
> [update|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/util/SqlShuttle.java#L117]
>
> happends
> {noformat}
> @Test void testSqlParserPosPlus() throws Exception {
> final String sql = "insert into emps select * from emps";
> SqlNode sqlNode = getSqlParser(sql).parseStmt();
> sqlNode.accept(new SqlShuttle(){
> @Override
> public SqlNode visit(SqlIdentifier identifier) {
> return new SqlIdentifier(identifier.names,
> identifier.getParserPosition());
> }
> });
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4019) Visit SqlInsert with SqlShuttle cause NullPointerException
[
https://issues.apache.org/jira/browse/CALCITE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119199#comment-17119199
]
Wang Yanlin commented on CALCITE-4019:
--
fixed in
[https://github.com/apache/calcite/commit/abe772018a6adb5007429e0c1c83b6e7d83a1c71],
thanks [~xzh_dz]
> Visit SqlInsert with SqlShuttle cause NullPointerException
> --
>
> Key: CALCITE-4019
> URL: https://issues.apache.org/jira/browse/CALCITE-4019
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When visit a SqlInsert with SqlShuttle, got NPE, the stack trace is
> {noformat}
> java.lang.NullPointerException
> at com.google.common.collect.Iterators$8.transform(Iterators.java:817)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at org.apache.calcite.sql.parser.SqlParserPos.sum(SqlParserPos.java:274)
> at
> org.apache.calcite.sql.parser.SqlParserPos.plusAll(SqlParserPos.java:183)
> at org.apache.calcite.sql.SqlOperator.createCall(SqlOperator.java:251)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.result(SqlShuttle.java:117)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.result(SqlShuttle.java:101)
> at org.apache.calcite.sql.util.SqlShuttle.visit(SqlShuttle.java:67)
> at org.apache.calcite.sql.util.SqlShuttle.visit(SqlShuttle.java:39)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
> at
> org.apache.calcite.sql.parser.SqlParserTest.testSqlParserPosPlus(SqlParserTest.java:8791)
> {noformat}
> Add this case in SqlParserTest to reproduce.
> return a new same SqlIdentifier for each SqlIdentifier, make sure
> [update|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/util/SqlShuttle.java#L117]
>
> happends
> {noformat}
> @Test void testSqlParserPosPlus() throws Exception {
> final String sql = "insert into emps select * from emps";
> SqlNode sqlNode = getSqlParser(sql).parseStmt();
> sqlNode.accept(new SqlShuttle(){
> @Override
> public SqlNode visit(SqlIdentifier identifier) {
> return new SqlIdentifier(identifier.names,
> identifier.getParserPosition());
> }
> });
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4019) Visit SqlInsert with SqlShuttle cause NullPointerException
[
https://issues.apache.org/jira/browse/CALCITE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yanlin resolved CALCITE-4019.
--
Resolution: Fixed
> Visit SqlInsert with SqlShuttle cause NullPointerException
> --
>
> Key: CALCITE-4019
> URL: https://issues.apache.org/jira/browse/CALCITE-4019
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When visit a SqlInsert with SqlShuttle, got NPE, the stack trace is
> {noformat}
> java.lang.NullPointerException
> at com.google.common.collect.Iterators$8.transform(Iterators.java:817)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at org.apache.calcite.sql.parser.SqlParserPos.sum(SqlParserPos.java:274)
> at
> org.apache.calcite.sql.parser.SqlParserPos.plusAll(SqlParserPos.java:183)
> at org.apache.calcite.sql.SqlOperator.createCall(SqlOperator.java:251)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.result(SqlShuttle.java:117)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.result(SqlShuttle.java:101)
> at org.apache.calcite.sql.util.SqlShuttle.visit(SqlShuttle.java:67)
> at org.apache.calcite.sql.util.SqlShuttle.visit(SqlShuttle.java:39)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
> at
> org.apache.calcite.sql.parser.SqlParserTest.testSqlParserPosPlus(SqlParserTest.java:8791)
> {noformat}
> Add this case in SqlParserTest to reproduce.
> return a new same SqlIdentifier for each SqlIdentifier, make sure
> [update|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/util/SqlShuttle.java#L117]
>
> happends
> {noformat}
> @Test void testSqlParserPosPlus() throws Exception {
> final String sql = "insert into emps select * from emps";
> SqlNode sqlNode = getSqlParser(sql).parseStmt();
> sqlNode.accept(new SqlShuttle(){
> @Override
> public SqlNode visit(SqlIdentifier identifier) {
> return new SqlIdentifier(identifier.names,
> identifier.getParserPosition());
> }
> });
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4029) ProjectRemoveRule auto pruning may prevent rules from running if mixed conventions are used in a logical plan
[
https://issues.apache.org/jira/browse/CALCITE-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Haidai updated CALCITE-4029:
--
Description:
Preconditions to reproduce the issue:
# Logical plan has mixed conventions (for example, a bottom node is a
TableScan in a final convention while other nodes are regular logical nodes
with NONE convention).
# There is a rule that expects a logical node with an input (like a rule
matching "operand(LogicalSort.class, operand(RelNode.class, any()))")
# A project over the scan is trivial (like SELECT * FROM ...)
The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939,
please see comments for a detailed debugging of a real-life reproducing case.
h4. Example:
Logical plan with a leaf nodes in a custom convention:
{code:java}
LogicalSort[NONE]
LogicalProject[NONE]
CustomScan[CUSTOM_CONVENTION]{code}
A rule configured (RuleX) matches "operand(LogicalSort.class,
operand(RelNode.class, any()))".
*Without ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial an merges it into a
single RelSet with CustomScan.
RuleX can run on top of this change as far as LogicalProject has a logical node
(LogicalProject in RelSubset[NONE]) as an input.
*With ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial but removes it with it's
RelSet so the CustomScan is the only node in it's RelSet,
RelSubset[CUSTOM_CONVENTION].
RuleX can't run on top of this change as far as LogicalProject has an empty
input RelSubset[NONE] of the RelSet with the CustomScan.
h2. Possible workarounds
# Disable ProjectRemoveRule auto pruning.
# Use only logical nodes in a logical plan, for the example above: use
LogicalScan - > CustomScanRule - > CustomScan instead of direct use of
CustomScan.
was:
Preconditions to reproduce the issue:
# Logical plan has mixed conventions (for example, a bottom node is a
TableScan in a final convention while other nodes are regular logical nodes
with NONE convention).
# There is a rule that expects a logical node with an input (like a rule
matching "operand(LogicalSort.class, operand(RelNode.class, any()))")
# A project over the scan is trivial (like SELECT * FROM ...)
The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939,
please see comments for a detailed debugging of a real-life reproducing case.
h4. Example:
Logical plan with a leaf nodes in a custom convention:
{code:java}
LogicalSort[NONE]
LogicalProject[NONE]
CustomScan[CUSTOM_CONVENTION]{code}
A rule configured (RuleX) matches "operand(LogicalSort.class,
operand(RelNode.class, any()))".
*Without ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial an merges it into a
single RelSet with CustomScan.
RuleX can run on top of this change as far as LogicalProject has a logical node
(LogicalProject in RelSubset[NONE]) as an input.
*With ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial but removes with it's
RelSet so the CustomScan is the only node in it's RelSet,
RelSubset[CUSTOM_CONVENTION].
RuleX can't run on top of this change as far as LogicalProject has an empty
input RelSubset[NONE] of the RelSet with the CustomScan.
h2. Possible workarounds
# Disable ProjectRemoveRule auto pruning.
# Use only logical nodes in a logical plan, for the example above: use
LogicalScan - > CustomScanRule - > CustomScan instead of direct use of
CustomScan.
> ProjectRemoveRule auto pruning may prevent rules from running if mixed
> conventions are used in a logical plan
> --
>
> Key: CALCITE-4029
> URL: https://issues.apache.org/jira/browse/CALCITE-4029
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Anton Haidai
>Priority: Minor
>
> Preconditions to reproduce the issue:
> # Logical plan has mixed conventions (for example, a bottom node is a
> TableScan in a final convention while other nodes are regular logical nodes
> with NONE convention).
> # There is a rule that expects a logical node with an input (like a rule
> matching "operand(LogicalSort.class, operand(RelNode.class, any()))")
> # A project over the scan is trivial (like SELECT * FROM ...)
> The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939,
> please see comments for a detailed debugging of a real-life reproducing case.
> h4. Example:
> Logical plan with a leaf nodes in a custom convention:
> {code:java}
> LogicalSort[NONE]
> LogicalProject[NONE]
> CustomScan[CUSTOM_CONVENTION]{code}
> A rule configured (RuleX) matches "operand(LogicalSort.class,
> operand(RelNode.class, any()))".
> *Without
[jira] [Updated] (CALCITE-4029) ProjectRemoveRule auto pruning may prevent rules from running if mixed conventions are used in a logical plan
[
https://issues.apache.org/jira/browse/CALCITE-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Haidai updated CALCITE-4029:
--
Description:
Preconditions to reproduce the issue:
# Logical plan has mixed conventions (for example, a bottom node is a
TableScan in a final convention while other nodes are regular logical nodes
with NONE convention).
# There is a rule that expects a logical node with an input (like a rule
matching "operand(LogicalSort.class, operand(RelNode.class, any()))")
# A project over the scan is trivial (like SELECT * FROM ...)
The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939,
please see comments for a detailed debugging of a real-life reproducing case.
h4. Example:
Logical plan with a leaf nodes in a custom convention:
{code:java}
LogicalSort[NONE]
LogicalProject[NONE]
CustomScan[CUSTOM_CONVENTION]{code}
A rule configured (RuleX) matches "operand(LogicalSort.class,
operand(RelNode.class, any()))".
*Without ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial an merges it into a
single RelSet with CustomScan.
RuleX can run on top of this change as far as LogicalProject has a logical node
(LogicalProject in RelSubset[NONE]) as an input.
*With ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial but removes with it's
RelSet so the CustomScan is the only node in it's RelSet,
RelSubset[CUSTOM_CONVENTION].
RuleX can't run on top of this change as far as LogicalProject has an empty
input RelSubset[NONE] of the RelSet with the CustomScan.
h2. Possible workarounds
# Disable ProjectRemoveRule auto pruning.
# Use only logical nodes in a logical plan, for the example above: use
LogicalScan - > CustomScanRule - > CustomScan instead of direct use of
CustomScan.
was:
Preconditions to reproduce the issue:
# Logical plan has mixed conventions (for example, a bottom node is a
TableScan in a final convention while other nodes are regular logical nodes
with NONE convention).
# There is a rule that expects a logical node with an input (like a rule
matching "operand(LogicalSort.class, operand(RelNode.class, any()))")
# A project over the scan is trivial (like SELECT * FROM ...)
The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939,
please see comments regarding a detailed debugging of a real-life reproducing
case.
h4. Example:
Logical plan with a leaf nodes in a custom convention:
LogicalSort[NONE]
LogicalProject[NONE]
CustomScan[CUSTOM_CONVENTION]
A rule configured (RuleX) matches "operand(LogicalSort.class,
operand(RelNode.class, any()))".
*Without ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial an merges it into a
single RelSet with CustomScan.
RuleX can run on top of this change as far as LogicalProject has a logical node
(LogicalProject in RelSubset[NONE]) as an input.
*With ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial but removes with it's
RelSet so the CustomScan is the only node in it's RelSet,
RelSubset[CUSTOM_CONVENTION].
RuleX can't run on top of this change as far as LogicalProject has an empty
input RelSubset[NONE] of the RelSet with the CustomScan.
h2. Possible workarounds
# Disable ProjectRemoveRule auto pruning.
# Use only logical nodes in a logical plan, for the example above: use
LogicalScan - > CustomScanRule - > CustomScan instead of direct use of
CustomScan.
> ProjectRemoveRule auto pruning may prevent rules from running if mixed
> conventions are used in a logical plan
> --
>
> Key: CALCITE-4029
> URL: https://issues.apache.org/jira/browse/CALCITE-4029
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Anton Haidai
>Priority: Minor
>
> Preconditions to reproduce the issue:
> # Logical plan has mixed conventions (for example, a bottom node is a
> TableScan in a final convention while other nodes are regular logical nodes
> with NONE convention).
> # There is a rule that expects a logical node with an input (like a rule
> matching "operand(LogicalSort.class, operand(RelNode.class, any()))")
> # A project over the scan is trivial (like SELECT * FROM ...)
> The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939,
> please see comments for a detailed debugging of a real-life reproducing case.
> h4. Example:
> Logical plan with a leaf nodes in a custom convention:
> {code:java}
> LogicalSort[NONE]
> LogicalProject[NONE]
> CustomScan[CUSTOM_CONVENTION]{code}
> A rule configured (RuleX) matches "operand(LogicalSort.class,
> operand(RelNode.class, any()))".
> *Without ProjectRemoveRule
[jira] [Updated] (CALCITE-4029) ProjectRemoveRule auto pruning may prevent rules from running if mixed conventions are used in a logical plan
[
https://issues.apache.org/jira/browse/CALCITE-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Haidai updated CALCITE-4029:
--
Description:
Preconditions to reproduce the issue:
# Logical plan has mixed conventions (for example, a bottom node is a
TableScan in a final convention while other nodes are regular logical nodes
with NONE convention).
# There is a rule that expects a logical node with an input (like a rule
matching "operand(LogicalSort.class, operand(RelNode.class, any()))")
# A project over the scan is trivial (like SELECT * FROM ...)
The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939,
please see comments for a detailed debugging of a real-life reproducing case.
h4. Example:
Logical plan with a leaf nodes in a custom convention:
{code:java}
LogicalSort[NONE]
LogicalProject[NONE]
CustomScan[CUSTOM_CONVENTION]{code}
A rule configured (RuleX) matches "operand(LogicalSort.class,
operand(RelNode.class, any()))".
*Without ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial an merges it into a
single RelSet with CustomScan.
RuleX can run on top of this change as far as LogicalProject has a logical node
(LogicalProject in RelSubset[NONE]) as an input.
*With ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial but removes with it's
RelSet so the CustomScan is the only node in it's RelSet,
RelSubset[CUSTOM_CONVENTION].
RuleX can't run on top of this change as far as LogicalProject has an empty
input RelSubset[NONE] of the RelSet with the CustomScan.
h2. Possible workarounds
# Disable ProjectRemoveRule auto pruning.
# Use only logical nodes in a logical plan, for the example above: use
LogicalScan - > CustomScanRule - > CustomScan instead of direct use of
CustomScan.
was:
Preconditions to reproduce the issue:
# Logical plan has mixed conventions (for example, a bottom node is a
TableScan in a final convention while other nodes are regular logical nodes
with NONE convention).
# There is a rule that expects a logical node with an input (like a rule
matching "operand(LogicalSort.class, operand(RelNode.class, any()))")
# A project over the scan is trivial (like SELECT * FROM ...)
The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939,
please see comments for a detailed debugging of a real-life reproducing case.
h4. Example:
Logical plan with a leaf nodes in a custom convention:
{code:java}
LogicalSort[NONE]
LogicalProject[NONE]
CustomScan[CUSTOM_CONVENTION]{code}
A rule configured (RuleX) matches "operand(LogicalSort.class,
operand(RelNode.class, any()))".
*Without ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial an merges it into a
single RelSet with CustomScan.
RuleX can run on top of this change as far as LogicalProject has a logical node
(LogicalProject in RelSubset[NONE]) as an input.
*With ProjectRemoveRule auto pruning*
ProjectRemoveRule recognizes LogicalProject as trivial but removes with it's
RelSet so the CustomScan is the only node in it's RelSet,
RelSubset[CUSTOM_CONVENTION].
RuleX can't run on top of this change as far as LogicalProject has an empty
input RelSubset[NONE] of the RelSet with the CustomScan.
h2. Possible workarounds
# Disable ProjectRemoveRule auto pruning.
# Use only logical nodes in a logical plan, for the example above: use
LogicalScan - > CustomScanRule - > CustomScan instead of direct use of
CustomScan.
> ProjectRemoveRule auto pruning may prevent rules from running if mixed
> conventions are used in a logical plan
> --
>
> Key: CALCITE-4029
> URL: https://issues.apache.org/jira/browse/CALCITE-4029
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Anton Haidai
>Priority: Minor
>
> Preconditions to reproduce the issue:
> # Logical plan has mixed conventions (for example, a bottom node is a
> TableScan in a final convention while other nodes are regular logical nodes
> with NONE convention).
> # There is a rule that expects a logical node with an input (like a rule
> matching "operand(LogicalSort.class, operand(RelNode.class, any()))")
> # A project over the scan is trivial (like SELECT * FROM ...)
> The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939,
> please see comments for a detailed debugging of a real-life reproducing case.
> h4. Example:
> Logical plan with a leaf nodes in a custom convention:
> {code:java}
> LogicalSort[NONE]
> LogicalProject[NONE]
> CustomScan[CUSTOM_CONVENTION]{code}
> A rule configured (RuleX) matches "operand(LogicalSort.class,
> operand(RelNode.class, any()))".
> *Without
[jira] [Commented] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[ https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119134#comment-17119134 ] Anton Haidai commented on CALCITE-3939: --- Here it is: https://issues.apache.org/jira/browse/CALCITE-4029 > Change UnionEliminatorRule and ProjectRemoveRule to auto pruning > SubstitutionRule > - > > Key: CALCITE-3939 > URL: https://issues.apache.org/jira/browse/CALCITE-3939 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a > RelNode. They can also become SubstitutionRule with autoprune enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4029) ProjectRemoveRule auto pruning may prevent rules from running if mixed conventions are used in a logical plan
Anton Haidai created CALCITE-4029: - Summary: ProjectRemoveRule auto pruning may prevent rules from running if mixed conventions are used in a logical plan Key: CALCITE-4029 URL: https://issues.apache.org/jira/browse/CALCITE-4029 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.23.0 Reporter: Anton Haidai Preconditions to reproduce the issue: # Logical plan has mixed conventions (for example, a bottom node is a TableScan in a final convention while other nodes are regular logical nodes with NONE convention). # There is a rule that expects a logical node with an input (like a rule matching "operand(LogicalSort.class, operand(RelNode.class, any()))") # A project over the scan is trivial (like SELECT * FROM ...) The issue is related to https://issues.apache.org/jira/browse/CALCITE-3939, please see comments regarding a detailed debugging of a real-life reproducing case. h4. Example: Logical plan with a leaf nodes in a custom convention: LogicalSort[NONE] LogicalProject[NONE] CustomScan[CUSTOM_CONVENTION] A rule configured (RuleX) matches "operand(LogicalSort.class, operand(RelNode.class, any()))". *Without ProjectRemoveRule auto pruning* ProjectRemoveRule recognizes LogicalProject as trivial an merges it into a single RelSet with CustomScan. RuleX can run on top of this change as far as LogicalProject has a logical node (LogicalProject in RelSubset[NONE]) as an input. *With ProjectRemoveRule auto pruning* ProjectRemoveRule recognizes LogicalProject as trivial but removes with it's RelSet so the CustomScan is the only node in it's RelSet, RelSubset[CUSTOM_CONVENTION]. RuleX can't run on top of this change as far as LogicalProject has an empty input RelSubset[NONE] of the RelSet with the CustomScan. h2. Possible workarounds # Disable ProjectRemoveRule auto pruning. # Use only logical nodes in a logical plan, for the example above: use LogicalScan - > CustomScanRule - > CustomScan instead of direct use of CustomScan. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[ https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119117#comment-17119117 ] Haisheng Yuan commented on CALCITE-3939: [~anha] Thanks for the detailed description. Can you create a new issue? So that we can continue discussion there. Thanks. > Change UnionEliminatorRule and ProjectRemoveRule to auto pruning > SubstitutionRule > - > > Key: CALCITE-3939 > URL: https://issues.apache.org/jira/browse/CALCITE-3939 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a > RelNode. They can also become SubstitutionRule with autoprune enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[ https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119058#comment-17119058 ] Anton Haidai commented on CALCITE-3939: --- [~julianhyde] , sorry, it took long to compose the previous comment so I didn't see your comment before submitting. I'm still not sure if this is a bug an if this is something that could affect other Calcite's users. To summarize, if my conclusions are correct, changes in the scope of this task may lead to ignored rules on logical nodes with input if non-logical nodes (like non-logical table scans) are used in logical plans. If it seems like a bug for Calcite's developers, please feel free to create a new issue regarding it or ping me if I should create this new issue. > Change UnionEliminatorRule and ProjectRemoveRule to auto pruning > SubstitutionRule > - > > Key: CALCITE-3939 > URL: https://issues.apache.org/jira/browse/CALCITE-3939 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a > RelNode. They can also become SubstitutionRule with autoprune enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[
https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119048#comment-17119048
]
Anton Haidai commented on CALCITE-3939:
---
[~botong], thank you for you detailed description. I did some debugging and I
think that I figured out what is going on, so here are debugging results
aligned with the steps from your message.
h2. Step 1
happens as described.
h2. Step 2
Due to the autoPruneOld() returning true, there is a call
volcanoPlanner.prune() before the merge of and
.
The merge itself: there is a call .mergeWith()
So the is being removed from a list of sets.
There is an attempt to add into :
planner.reregister(, );
But this VolcanoPlanner.reregister() method has the following check:
{code:java}
if (!prunedNodes.contains(rel)) {
addRelToSet(rel, set);
}{code}
So there is a difference with your scenario: instead of both
and being in , there is only there while the
is just thrown away.
h2. Step 3
This never happens. I think, this is because
obviously has a convention NONE as a logical node, but is *not* a
logical node and has some final custom convention from the beginning. So while
the is the only node in a set, there is no other nodes with NONE
convention in this set so input is an empty subset with NONE
convention and can't match once more and never happens.
h2. Step 4
all as described, is ignored (RuleQueue.skipMatch() method).
h2. Step 5
As described in Step 3 never happens.
My workaround (force ProjectRemoveRule.autoPruneOld() to return false) make it
work exactly as you described: contains both and
making possible because has some
input with NONE convention.
Another workaround could be using something like [NONE] ->
-> [FINAL_CONVENTION] instead of a direct use of
[FINAL_CONVENTION] in a logical plan: pretty sure it will work, but
I didn't check it.
> Change UnionEliminatorRule and ProjectRemoveRule to auto pruning
> SubstitutionRule
> -
>
> Key: CALCITE-3939
> URL: https://issues.apache.org/jira/browse/CALCITE-3939
> Project: Calcite
> Issue Type: Improvement
>Reporter: Botong Huang
>Priority: Major
> Fix For: 1.23.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a
> RelNode. They can also become SubstitutionRule with autoprune enabled
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3094) Code of method grows beyond 64 KB
[
https://issues.apache.org/jira/browse/CALCITE-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119040#comment-17119040
]
Julian Hyde commented on CALCITE-3094:
--
It might not be easy to convince the code generator replace that code with
calls to {{System.arraycopy}}. And even if we did, I think it's likely that
another piece of repetitious code would push us over the limit.
> Code of method grows beyond 64 KB
> -
>
> Key: CALCITE-3094
> URL: https://issues.apache.org/jira/browse/CALCITE-3094
> Project: Calcite
> Issue Type: Bug
> Components: core, jdbc-adapter
>Affects Versions: 1.19.0
>Reporter: Pablo
>Priority: Critical
>
> I am running simple queries like these:
> {{SELECT * FROM t0 INNER JOIN t1 ON t0.`f00` = t1.`f03`"}}
>
> They are running ok, but, when t1 and t0 have about 3290 fields, I get this
> error.
> Is there anything I can do? I can't do joins with this amount of fields?
>
> Thank you in advance
>
> {noformat}
> Caused by: java.lang.RuntimeException: Error while compiling generated Java
> code:
> public org.apache.calcite.linq4j.Enumerable bind(final
> org.apache.calcite.DataContext root) {
> final org.apache.calcite.rel.RelNode v2stashed =
> (org.apache.calcite.rel.RelNode) root.get("v2stashed");
> final org.apache.calcite.rel.RelNode v1stashed =
> (org.apache.calcite.rel.RelNode) root.get("v1stashed");
> final org.apache.calcite.interpreter.Interpreter interpreter = new
> org.apache.calcite.interpreter.Interpreter(
> root,
> v1stashed);
> final org.apache.calcite.interpreter.Interpreter interpreter0 = new
> org.apache.calcite.interpreter.Interpreter(
> root,
> v2stashed);
> return interpreter.join(interpreter0, new
> org.apache.calcite.linq4j.function.Function1() {
> public Byte apply(Object[] v1) {
> return (Byte) v1[0];
> }
> public Object apply(Object v1) {
> return apply(
> (Object[]) v1);
> }
> }
> , new org.apache.calcite.linq4j.function.Function1() {
> public Byte apply(Object[] v1) {
> return (Byte) v1[3];
> }
> public Object apply(Object v1) {
> return apply(
> (Object[]) v1);
> }
> }
> , new org.apache.calcite.linq4j.function.Function2() {
> public Object[] apply(Object[] left, Object[] right) {
> return new Object[] {
> left[0],
> left[1],
> left[2],
> [...elided ...]
> left[3298],
> left[3299],
> right[0],
> right[1],
> [...elided...]]
> right[3295],
> right[3296],
> right[3297],
> right[3298],
> right[3299]};
> }
> public Object[] apply(Object left, Object right) {
> return apply(
> (Object[]) left,
> (Object[]) right);
> }
> }
> , null, false, false);
> }
> public Class getElementType() {
> return java.lang.Object[].class;
> }
> at org.apache.calcite.avatica.Helper.wrap(Helper.java:37)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:128)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1233)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:332)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:767)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:631)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:601)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:229)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:211)
> ... 80 more
> Caused by: org.codehaus.janino.InternalCompilerException: Compiling "Baz":
> Code of method
> "apply([Ljava/lang/Object;[Ljava/lang/Object;)[Ljava/lang/Object;" of class
> "Baz$3" grows beyond 64 KB
> at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> at
> org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:313)
> at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:235)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:50)
> at
> org.codehaus.janino.ClassBodyEvaluator.createInstance(ClassBodyEvaluator.java:347)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.getBindable(EnumerableInterpretable.java:162)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:125)
> ... 88 more
> Caused by: org.codehaus.janino.InternalCompilerException: Code of method
> "apply([Ljava/lang/Object;[Ljava/lang/Object;)[Ljava/lang/Object;" of class
> "Baz$3"
[jira] [Commented] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[ https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119034#comment-17119034 ] Julian Hyde commented on CALCITE-3939: -- Should this discussion be in a new JIRA case? If yes, please post a link to the new case and continue discussion there. > Change UnionEliminatorRule and ProjectRemoveRule to auto pruning > SubstitutionRule > - > > Key: CALCITE-3939 > URL: https://issues.apache.org/jira/browse/CALCITE-3939 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a > RelNode. They can also become SubstitutionRule with autoprune enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[ https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118941#comment-17118941 ] Botong Huang commented on CALCITE-3939: --- Thanks [~anha] for reporting the issue. However, as [~hyuan] mentioned, something is missing in your scenario. Let's say the LogicalSort, TrivialProject, CustomScan are in RelSet 1, 2, 3 respectively. Here's what it should have happened. # RuleX matches LogicalSort and TrivialProject (RuleX.matches called with ). This RuleMatch1 gets queued in RuleQueue. # ProjectRemoveRule fires, it marks TrivialProject as stale, triggers RelSet 2 and 3 to merge, say into RelSet 2. Now both TrivialProject and CustomScan are in RelSet 2. # The planner.fireRules in RelSet.mergeWith will trigger a new round of RuleMatch for all relNodes in RelSet 2. Specifically RuleX should have a new match with LogicalSort and CustomScan (RuleX.matches called with ). This RuleMatch2 is also queued in RuleQueue. # RuleMatch1 pops from ruleQueue, and since TrivialProject is stale, this match is skipped and onMatch not called. # RuleMatch2 pops from ruleQueue, and RuleX.onMatch called with . Please let us know what went wrong here. > Change UnionEliminatorRule and ProjectRemoveRule to auto pruning > SubstitutionRule > - > > Key: CALCITE-3939 > URL: https://issues.apache.org/jira/browse/CALCITE-3939 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a > RelNode. They can also become SubstitutionRule with autoprune enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[
https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118796#comment-17118796
]
Anton Haidai commented on CALCITE-3939:
---
{quote}
Does RuleX look into the siblings of Project in the RelSet through RelSubset?
{quote}
No, just a couple of application-specific instanceofs on call.rel(0). Even
more, I was trying to move all logic into RuleX.onMatch() levaing
RuleX.matches() trivial:
{code:java}
@Override
public boolean matches(RelOptRuleCall call) {
final Sort sort = call.rel(0);
final RelNode node = call.rel(1);
return true;
}{code}
And everything still works according to the description. With the workaround,
there are numerous calls of RuleX.matches() and RuleX.onMatch() but without it,
there is only a single call of RuleX.matches() which returns true and zero
calls of RuleX.onMatch() (as described above).
> Change UnionEliminatorRule and ProjectRemoveRule to auto pruning
> SubstitutionRule
> -
>
> Key: CALCITE-3939
> URL: https://issues.apache.org/jira/browse/CALCITE-3939
> Project: Calcite
> Issue Type: Improvement
>Reporter: Botong Huang
>Priority: Major
> Fix For: 1.23.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a
> RelNode. They can also become SubstitutionRule with autoprune enabled
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (CALCITE-4028) Unknown error in simple query from JDBCSchema connection to MS SqlServer
[
https://issues.apache.org/jira/browse/CALCITE-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Hait closed CALCITE-4028.
---
Resolution: Not A Bug
> Unknown error in simple query from JDBCSchema connection to MS SqlServer
>
>
> Key: CALCITE-4028
> URL: https://issues.apache.org/jira/browse/CALCITE-4028
> Project: Calcite
> Issue Type: Bug
> Components: jdbc-adapter
>Affects Versions: 1.23.0
>Reporter: David Hait
>Priority: Major
>
> I tried to replicate the "Background" example, but with MSSQL instead of
> MySql. This is failing and it is not obvious why - the exception and message
> do not give enough information.
> {code:java}
> BasicDataSource dataSource = new BasicDataSource();
> dataSource.setUrl("jdbc:sqlserver://tsql202.dc.optionmetrics.com");
> dataSource.setUsername("");
> dataSource.setPassword("*");
> dataSource.setDefaultCatalog("ivydb");
> dataSource.setDefaultSchema("dbo");
> Schema schema = JdbcSchema.create(rootSchema,"sql",dataSource,null,null);
> rootSchema.add("dbo",schema);
> Statement statement = calciteConnection.createStatement();
> var rs2 = statement.executeQuery("SELECT SecurityID from dbo.SECURITY");
> System.out.println(rs2);
> {code}
> This gave an undecipherable exception:
> (it's got that "Baz.bind" thing):
>
> {noformat}
> java.sql.SQLException: Error while executing SQL "SELECT SecurityID from
> dbo.SECURITY": nulljava.sql.SQLException: Error while executing SQL "SELECT
> SecurityID from dbo.SECURITY": null
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
> at
> com.optionmetrics.adamo.AdamoApplicationTests.calciteDriverTest(AdamoApplicationTests.java:41)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:212)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:208)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
>
[jira] [Commented] (CALCITE-4028) Unknown error in simple query from JDBCSchema connection to MS SqlServer
[
https://issues.apache.org/jira/browse/CALCITE-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118791#comment-17118791
]
David Hait commented on CALCITE-4028:
-
A few minor tweaks to the catalog and schema, and now it works:
{{BasicDataSource dataSource = new BasicDataSource();}}
{{dataSource.setUrl("jdbc:sqlserver://tsql202.dc.optionmetrics.com;");}}
{{dataSource.setUsername("");}}
{{dataSource.setPassword("");}}
{{dataSource.setDefaultCatalog("ivydb");}}
{{Schema schema = JdbcSchema.create(rootSchema,"dbo",dataSource, null, "dbo");}}
{{rootSchema.add("dbo",schema);}}
Thank you for your guidance.
> Unknown error in simple query from JDBCSchema connection to MS SqlServer
>
>
> Key: CALCITE-4028
> URL: https://issues.apache.org/jira/browse/CALCITE-4028
> Project: Calcite
> Issue Type: Bug
> Components: jdbc-adapter
>Affects Versions: 1.23.0
>Reporter: David Hait
>Priority: Major
>
> I tried to replicate the "Background" example, but with MSSQL instead of
> MySql. This is failing and it is not obvious why - the exception and message
> do not give enough information.
> {code:java}
> BasicDataSource dataSource = new BasicDataSource();
> dataSource.setUrl("jdbc:sqlserver://tsql202.dc.optionmetrics.com");
> dataSource.setUsername("");
> dataSource.setPassword("*");
> dataSource.setDefaultCatalog("ivydb");
> dataSource.setDefaultSchema("dbo");
> Schema schema = JdbcSchema.create(rootSchema,"sql",dataSource,null,null);
> rootSchema.add("dbo",schema);
> Statement statement = calciteConnection.createStatement();
> var rs2 = statement.executeQuery("SELECT SecurityID from dbo.SECURITY");
> System.out.println(rs2);
> {code}
> This gave an undecipherable exception:
> (it's got that "Baz.bind" thing):
>
> {noformat}
> java.sql.SQLException: Error while executing SQL "SELECT SecurityID from
> dbo.SECURITY": nulljava.sql.SQLException: Error while executing SQL "SELECT
> SecurityID from dbo.SECURITY": null
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
> at
> com.optionmetrics.adamo.AdamoApplicationTests.calciteDriverTest(AdamoApplicationTests.java:41)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:212)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:208)
> at
>
[jira] [Commented] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[ https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118780#comment-17118780 ] Haisheng Yuan commented on CALCITE-3939: Thanks for reporting the issue, [~anha]. [~botong] Can you take a look? I am wondering why CustomScan doesn't trigger RuleX after set merge. Does RuleX look into the siblings of Project in the RelSet through RelSubset? > Change UnionEliminatorRule and ProjectRemoveRule to auto pruning > SubstitutionRule > - > > Key: CALCITE-3939 > URL: https://issues.apache.org/jira/browse/CALCITE-3939 > Project: Calcite > Issue Type: Improvement >Reporter: Botong Huang >Priority: Major > Fix For: 1.23.0 > > Time Spent: 50m > Remaining Estimate: 0h > > UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a > RelNode. They can also become SubstitutionRule with autoprune enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3939) Change UnionEliminatorRule and ProjectRemoveRule to auto pruning SubstitutionRule
[
https://issues.apache.org/jira/browse/CALCITE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118720#comment-17118720
]
Anton Haidai commented on CALCITE-3939:
---
FYI:
This change caused a regression on our side (with a trivial fix available).
I can't provide the test reproducing it as far as a custom rule (RuleX in a
description next) is involved, but I can describe the flow.
Preconditions:
RuleX is configured to match operand(LogicalSort.class, operand(RelNode.class,
any())).
Logical plan is:
{code:java}
LogicalSort(fetch=[5])
LogicalProject(id=[$0], name=[$1], customer_type=[$2], address_state=[$3],
address_city=[$4], zip_code=[$5], birth_date=[$6], registration_date=[$7],
registration_date_year=[$8], last_purchase_date=[$9])
CustomScan(){code}
This LogicalProject is trivial.
Flow:
# RuleX.matches() returns true, call nodes are LogicalSort, LogicalProject
# ProjectRemoveRule.onMatch() is called on LogicalProject and removes the
trivial project.
# RuleX.onMatch() is never called while it can produce a desirable
transformation: there are other nodes in the same RelSet with the removed
LogicalProject.
The trivial fix we used is just to make ProjectRemoveRule.autoPruneOld() to
return false reverting this fix.
I'm not sure if this is a bug and we have a workaround so this is just FYI.
> Change UnionEliminatorRule and ProjectRemoveRule to auto pruning
> SubstitutionRule
> -
>
> Key: CALCITE-3939
> URL: https://issues.apache.org/jira/browse/CALCITE-3939
> Project: Calcite
> Issue Type: Improvement
>Reporter: Botong Huang
>Priority: Major
> Fix For: 1.23.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> UnionEliminatorRule and ProjectRemoveRule are both pruning rules for a
> RelNode. They can also become SubstitutionRule with autoprune enabled
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3094) Code of method grows beyond 64 KB
[
https://issues.apache.org/jira/browse/CALCITE-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118612#comment-17118612
]
Thomas Rebele commented on CALCITE-3094:
In this particular case the problem seems to be
{code:java}
public Object[] apply(Object[] left, Object[] right) {
return new Object[] {
left[0],
left[1],
[...elided...]
left[3298],
left[3299],
right[0],
right[1],
[...elided...]
right[3298],
right[3299]};
}
{code}
Two calls to System.arraycopy(src, srcPos, dest, destPos, length) could do the
same and are probably more efficient.We could keep the current code if the
length of the new object < a constant (maybe 20?), and otherwise use
System.arraycopy(...).
> Code of method grows beyond 64 KB
> -
>
> Key: CALCITE-3094
> URL: https://issues.apache.org/jira/browse/CALCITE-3094
> Project: Calcite
> Issue Type: Bug
> Components: core, jdbc-adapter
>Affects Versions: 1.19.0
>Reporter: Pablo
>Priority: Critical
>
> I am running simple queries like these:
> {{SELECT * FROM t0 INNER JOIN t1 ON t0.`f00` = t1.`f03`"}}
>
> They are running ok, but, when t1 and t0 have about 3290 fields, I get this
> error.
> Is there anything I can do? I can't do joins with this amount of fields?
>
> Thank you in advance
>
> {noformat}
> Caused by: java.lang.RuntimeException: Error while compiling generated Java
> code:
> public org.apache.calcite.linq4j.Enumerable bind(final
> org.apache.calcite.DataContext root) {
> final org.apache.calcite.rel.RelNode v2stashed =
> (org.apache.calcite.rel.RelNode) root.get("v2stashed");
> final org.apache.calcite.rel.RelNode v1stashed =
> (org.apache.calcite.rel.RelNode) root.get("v1stashed");
> final org.apache.calcite.interpreter.Interpreter interpreter = new
> org.apache.calcite.interpreter.Interpreter(
> root,
> v1stashed);
> final org.apache.calcite.interpreter.Interpreter interpreter0 = new
> org.apache.calcite.interpreter.Interpreter(
> root,
> v2stashed);
> return interpreter.join(interpreter0, new
> org.apache.calcite.linq4j.function.Function1() {
> public Byte apply(Object[] v1) {
> return (Byte) v1[0];
> }
> public Object apply(Object v1) {
> return apply(
> (Object[]) v1);
> }
> }
> , new org.apache.calcite.linq4j.function.Function1() {
> public Byte apply(Object[] v1) {
> return (Byte) v1[3];
> }
> public Object apply(Object v1) {
> return apply(
> (Object[]) v1);
> }
> }
> , new org.apache.calcite.linq4j.function.Function2() {
> public Object[] apply(Object[] left, Object[] right) {
> return new Object[] {
> left[0],
> left[1],
> left[2],
> [...elided ...]
> left[3298],
> left[3299],
> right[0],
> right[1],
> [...elided...]]
> right[3295],
> right[3296],
> right[3297],
> right[3298],
> right[3299]};
> }
> public Object[] apply(Object left, Object right) {
> return apply(
> (Object[]) left,
> (Object[]) right);
> }
> }
> , null, false, false);
> }
> public Class getElementType() {
> return java.lang.Object[].class;
> }
> at org.apache.calcite.avatica.Helper.wrap(Helper.java:37)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:128)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1233)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:332)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:767)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:631)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:601)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:229)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:211)
> ... 80 more
> Caused by: org.codehaus.janino.InternalCompilerException: Compiling "Baz":
> Code of method
> "apply([Ljava/lang/Object;[Ljava/lang/Object;)[Ljava/lang/Object;" of class
> "Baz$3" grows beyond 64 KB
> at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> at
> org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:313)
> at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:235)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:50)
> at
> org.codehaus.janino.ClassBodyEvaluator.createInstance(ClassBodyEvaluator.java:347)
> at
>
[jira] [Commented] (CALCITE-4014) calcite convert(column using utf8mb4) throws exception
[ https://issues.apache.org/jira/browse/CALCITE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118445#comment-17118445 ] groobyming commented on CALCITE-4014: - Thanks [~julianhyde] > calcite convert(column using utf8mb4) throws exception > -- > > Key: CALCITE-4014 > URL: https://issues.apache.org/jira/browse/CALCITE-4014 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.21.0 >Reporter: groobyming >Priority: Major > Attachments: image-2020-05-20-17-00-28-611.png > > > Hello everyone, I found that when performing the following sql > verification, I will be prompted that the UTF8MB4 field cannot be found in > the table, but after reading the source code calcite is built-in to support > this convert function, is it not fully implemented yet? > > SQL is as follows: > select convert(name using utf8mb4) from test b where name = '4' > > Error Message: > !image-2020-05-20-17-00-28-611.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3951) Support different string comparison based on SqlCollation
[ https://issues.apache.org/jira/browse/CALCITE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118402#comment-17118402 ] Ruben Q L commented on CALCITE-3951: [~zabetak], 1.23 has been released and master is open, do you think we could move on with [PR#1937|https://github.com/apache/calcite/pull/1937]? Thanks. > Support different string comparison based on SqlCollation > - > > Key: CALCITE-3951 > URL: https://issues.apache.org/jira/browse/CALCITE-3951 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Fix For: 1.24.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Currently SqlCollation defines concepts like Coercibility, Charset, Locale, > etc. However, we cannot specify on a certain collation that e.g. a string > field should use case insensitive comparison. The goal of this ticket is to > evolve SqlCollation to support that, and adapt the corresponding classes to > use that (optional) "non-standard" comparison. -- This message was sent by Atlassian Jira (v8.3.4#803005)
