[jira] [Updated] (CALCITE-3173) RexNode Code Generation Problem
[ https://issues.apache.org/jira/browse/CALCITE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated CALCITE-3173: - Priority: Critical (was: Major) > RexNode Code Generation Problem > --- > > Key: CALCITE-3173 > URL: https://issues.apache.org/jira/browse/CALCITE-3173 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.20.0 >Reporter: Feng Zhu >Assignee: Feng Zhu >Priority: Critical > Labels: pull-request-available > Fix For: 1.21.0 > > Attachments: code.png, codegen.png > > > *Abstract:* Both RexImpTable and BlockBuilder have codegen issues, but it is > interesting that they can work together well for most cases. > We can illustrate the problem with a simple test case in JdbcTest, in > which the "commission" column is nullable. > {code:java} > @Test public void testNullSafeCheck() { > CalciteAssert.hr() > .query("select \"commission\" + 10 as s from \"hr\".\"emps\"") > .returns("S=1010\n" >+ "S=510\n" >+ "S=null\n" >+ "S=260\n"); > } > {code} > This test case can pass as the BlockBuilder is in default optimization > mode. However, when we set it into un-optimization mode in _EnumerableCalc_, > this test will fail due to NPE. The following picture demonstrates their > differences. > !code.png! > *1.RexImpTable generates unsafe code* > Before translating the RexCall (_*Add*_), RexImpTable firstly translate > its operands with (nullAs=*IS_NULL*) [1] to detect whether it is null (i.e., > {color:#ff}_inp4_unboxed_{color}). Then RexImpTable sets this operand's > nullable in RexToLixTranslator as {color:#FF}false{color} [2]. After > that, the operand will be translated again with *NOT_POSSIBLE* [3] to get the > value (i.e., {color:#ff}_inp4_0_unboxed_{color}). In the end, the RexCall > is implemented by NotNullImplementor.However, it is not safe to conduct > operations like unboxing in the second translation phase. > *2.BlockBuiler optimization's semantic issue buries NPE* > BlockBuilder.optimize() changes the code semantic in this case. For > conditional-like clauses (if...else, ?:, etc), InlineVariableVisitor will > wrongly make variables inlined. > In general, they can work together for most cases. However, when some > special branch is triggered by query, the problem will be exposed. For > example, the NewExpression (_new java.math.BigDecimal_) in CALCITE-3143 > breaks the inline optimization phase. > > *How to fix?* > I have digged into this problem a couple of days and tried many > approaches to fix it. But in this progress, I found the limitation in current > implementation. The whole recursive framework essentially conducts a > sequential codegen beheavior, and may visit a RexNode again and again with > different NullAs settings. > Due to the limitation, it is difficult to implement null-safe codegen > semantics with branching logic. We can also find that there are many branches > for special cases in current implementation. Even we can handle potential > issues every time, the logic will becomes more and more complex and > unfriendly for maintenance. > > Therefore, I propose to re-consider this part, including several initial > points. > (1) {color:#ff}_Visitor Pattern_{color} (RexVisitor). > Theoretically, RexNode can be translated into Expression by visiting the node > only once. We can implement RexVisitor rather than current recursive > translation. > (2)The {color:#ff}Result{color} consists of three items (code: > BuilderStatement, isNull: ParameterExpression, value: Expression).So it is > easy to decide how to implement a RexNode according to its children. > > Please correct me if I make something wrong. Look forward to suggestions! > > [1][https://github.com/apache/calcite/blob/1748f0503e7b626a8d0165f1698adb8b61bbc31e/core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java#L1062] > [2][https://github.com/apache/calcite/blob/1748f0503e7b626a8d0165f1698adb8b61bbc31e/core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java#L1064] > [3][https://github.com/apache/calcite/blob/1748f0503e7b626a8d0165f1698adb8b61bbc31e/core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java#L1113] > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (CALCITE-2302) Implicit type cast support
[ https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated CALCITE-2302: - Priority: Critical (was: Major) > Implicit type cast support > -- > > Key: CALCITE-2302 > URL: https://issues.apache.org/jira/browse/CALCITE-2302 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.17.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Critical > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive. > Implicit type cast is an useful function for many cases, So we should support > this. > I checkout Calcite code and found that: > # Now we use a validator to validate our operands types[ through kinds of > namespaces and scopes ] > # Most of the validations will finally goes to > {code:java} > SqlOperator.validateOperands > {code} > # which will use validation logic defined in corresponding > SqlOperandTypeChecker > What i'm confused about is where should i put the implicit type cast logic > in? I figured out 2 ways: > # Supply a tool class/rules to add casts into a parsed SqlNode tree which > will then go through the validation logic later on. > # Unleash the validation logic in kinds of SqlOperandTypeChecker, then > modify the RelNode/RexNodes tree converted from a validated SqlNode tree to > add in casts through custom RelOptRules. > So guys, which of the 2 ways should i go, or if there are better way to do > this? > I need your help. > > Updated 18-05-30: > Hi guys, i have made a PR in > [CALCITE-2302|https://github.com/apache/calcite/pull/706] > This is design doc: [Calcite Implicit Type Cast > Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing]. > This is the conversion types mapping: [Conversion Types > Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing]. > I really appreciate your suggestions, thx. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-2302) Implicit type cast support
[ https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909601#comment-16909601 ] Stamatis Zampetakis commented on CALCITE-2302: -- I think we agreed that it should be possible to enable/disable all implicit casts on demand; if that's the case then in principle we shouldn't break any of the existing queries in Postgres and other DBMS if the casts are disabled. If we are afraid that there are many cases that this may happen then we could choose to have type casts disabled by default for now. > Implicit type cast support > -- > > Key: CALCITE-2302 > URL: https://issues.apache.org/jira/browse/CALCITE-2302 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.17.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive. > Implicit type cast is an useful function for many cases, So we should support > this. > I checkout Calcite code and found that: > # Now we use a validator to validate our operands types[ through kinds of > namespaces and scopes ] > # Most of the validations will finally goes to > {code:java} > SqlOperator.validateOperands > {code} > # which will use validation logic defined in corresponding > SqlOperandTypeChecker > What i'm confused about is where should i put the implicit type cast logic > in? I figured out 2 ways: > # Supply a tool class/rules to add casts into a parsed SqlNode tree which > will then go through the validation logic later on. > # Unleash the validation logic in kinds of SqlOperandTypeChecker, then > modify the RelNode/RexNodes tree converted from a validated SqlNode tree to > add in casts through custom RelOptRules. > So guys, which of the 2 ways should i go, or if there are better way to do > this? > I need your help. > > Updated 18-05-30: > Hi guys, i have made a PR in > [CALCITE-2302|https://github.com/apache/calcite/pull/706] > This is design doc: [Calcite Implicit Type Cast > Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing]. > This is the conversion types mapping: [Conversion Types > Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing]. > I really appreciate your suggestions, thx. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909599#comment-16909599 ] Stamatis Zampetakis commented on CALCITE-3257: -- This is exactly the same problem with CALCITE-2018. It would be better not to disperse the discussion into multiple JIRAs; let's close this as duplicate and discuss under CALCITE-2018. > Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz > > > Key: CALCITE-3257 > URL: https://issues.apache.org/jira/browse/CALCITE-3257 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Xiening Dai >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > To repro - > 1. Set Log4J log level to TRACE. So planner will dump rel node info every > time at rule match > (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) > 2. Run JdbcTest.testNotExistsCorrelated. Get below exception - > java.lang.AssertionError: rel > [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has > lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 > cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]] > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) > at > org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) > at > org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) > at > org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556) > at > org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod
[jira] [Updated] (CALCITE-3259) Align 'Property' in the serialized xml string of RelXmlWriter.
[ https://issues.apache.org/jira/browse/CALCITE-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-3259: Labels: pull-request-available (was: ) > Align 'Property' in the serialized xml string of RelXmlWriter. > -- > > Key: CALCITE-3259 > URL: https://issues.apache.org/jira/browse/CALCITE-3259 > Project: Calcite > Issue Type: Improvement >Reporter: Wang Yanlin >Priority: Minor > Labels: pull-request-available > > In the serialized xml string of RelXmlWriter, the 'Property' label is > aligned. > For the sql below > {noformat} > select 1 + 2, 3 from (values (true)) > {noformat} > the serialized xml string is like this: > {noformat} > > > +(1, 2) > > 3 > > > > [{ true }] > > > > > {noformat} > It's better to make 'Property' aligned. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (CALCITE-3259) Align 'Property' in the serialized xml string of RelXmlWriter.
Wang Yanlin created CALCITE-3259: Summary: Align 'Property' in the serialized xml string of RelXmlWriter. Key: CALCITE-3259 URL: https://issues.apache.org/jira/browse/CALCITE-3259 Project: Calcite Issue Type: Improvement Reporter: Wang Yanlin In the serialized xml string of RelXmlWriter, the 'Property' label is aligned. For the sql below {noformat} select 1 + 2, 3 from (values (true)) {noformat} the serialized xml string is like this: {noformat} +(1, 2) 3 [{ true }] {noformat} It's better to make 'Property' aligned. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view
[ https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing reassigned CALCITE-3226: - Assignee: jin xing > RelBuilder doesn't keep the alias when scan from an expanded view > - > > Key: CALCITE-3226 > URL: https://issues.apache.org/jira/browse/CALCITE-3226 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: jin xing >Assignee: jin xing >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In current implementation of {{RelBuilder::scan}} > ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]), > the alias can be derived and recorded into {{Frame}} only when the > {{RelNode}} is a {{TableScan}} > ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]). > But when {{RelBuilder::scan}} from an expanded view, the node is not a > {{TableScan}} and the alias is not kept. Below test failed - we cannot > reference a field by alias "MYVIEW". > {code:java} > @Test public void testExpandViewShouldKeepAlias() throws SQLException { > try (Connection connection = > DriverManager.getConnection("jdbc:calcite:")) { > final Frameworks.ConfigBuilder configBuilder = > expandingConfig(connection); > final RelOptTable.ViewExpander viewExpander = > (RelOptTable.ViewExpander) > Frameworks.getPlanner(configBuilder.build()); > final RelFactories.TableScanFactory tableScanFactory = > RelFactories.expandingScanFactory(viewExpander, > RelFactories.DEFAULT_TABLE_SCAN_FACTORY); > configBuilder.context(Contexts.of(tableScanFactory)); > final RelBuilder builder = RelBuilder.create(configBuilder.build()); > RelNode node = > builder.scan("MYVIEW") > .project( > builder.field(1, "MYVIEW", "EMPNO"), // Exception > thrown from here. > builder.field(1, "MYVIEW", "ENAME")) > .build(); > String expected = > "LogicalProject(EMPNO=[$0], ENAME=[$1])\n" > + " LogicalFilter(condition=[=(1, 1)])\n" > + "LogicalTableScan(table=[[scott, EMP]])\n"; > assertThat(node, hasTree(expected)); > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view
[ https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909567#comment-16909567 ] jin xing edited comment on CALCITE-3226 at 8/17/19 4:06 AM: Thanks a lot for clarify [~rubenql] I think [~julianhyde] expects to add a test case where the view of “select * from t” expands as a Scan, but I tried and turned out that it always expands as a Project+Scan. For a common table scan, the alias would have already been correctly set and I added a test case testProjectWithAliasFromScan(https://github.com/apache/calcite/pull/1351/files#diff-9f6a8f13d9ad70d7336340bf6b2e7b71R702) Thanks a lot for review this issue [~julianhyde] [~rubenql] and sorry for my late reply was (Author: jinxing6...@126.com): Thanks a lot for clarify [~rubenql] I think [~julianhyde] expects to add a test case where the view of “select * from t” expands as a Scan, but I tried and turned out that it always expands as a Project+Scan. For a common table scan, the alias would have already been correctly set and I added a test case testProjectWithAliasFromScan(https://github.com/apache/calcite/pull/1351/files#diff-9f6a8f13d9ad70d7336340bf6b2e7b71R702) Thanks a lot for review this issue [~julianhyde][~rubenql] and sorry for my late reply > RelBuilder doesn't keep the alias when scan from an expanded view > - > > Key: CALCITE-3226 > URL: https://issues.apache.org/jira/browse/CALCITE-3226 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: jin xing >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In current implementation of {{RelBuilder::scan}} > ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]), > the alias can be derived and recorded into {{Frame}} only when the > {{RelNode}} is a {{TableScan}} > ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]). > But when {{RelBuilder::scan}} from an expanded view, the node is not a > {{TableScan}} and the alias is not kept. Below test failed - we cannot > reference a field by alias "MYVIEW". > {code:java} > @Test public void testExpandViewShouldKeepAlias() throws SQLException { > try (Connection connection = > DriverManager.getConnection("jdbc:calcite:")) { > final Frameworks.ConfigBuilder configBuilder = > expandingConfig(connection); > final RelOptTable.ViewExpander viewExpander = > (RelOptTable.ViewExpander) > Frameworks.getPlanner(configBuilder.build()); > final RelFactories.TableScanFactory tableScanFactory = > RelFactories.expandingScanFactory(viewExpander, > RelFactories.DEFAULT_TABLE_SCAN_FACTORY); > configBuilder.context(Contexts.of(tableScanFactory)); > final RelBuilder builder = RelBuilder.create(configBuilder.build()); > RelNode node = > builder.scan("MYVIEW") > .project( > builder.field(1, "MYVIEW", "EMPNO"), // Exception > thrown from here. > builder.field(1, "MYVIEW", "ENAME")) > .build(); > String expected = > "LogicalProject(EMPNO=[$0], ENAME=[$1])\n" > + " LogicalFilter(condition=[=(1, 1)])\n" > + "LogicalTableScan(table=[[scott, EMP]])\n"; > assertThat(node, hasTree(expected)); > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view
[ https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909567#comment-16909567 ] jin xing commented on CALCITE-3226: --- Thanks a lot for clarify [~rubenql] I think [~julianhyde] expects to add a test case where the view of “select * from t” expands as a Scan, but I tried and turned out that it always expands as a Project+Scan. For a common table scan, the alias would have already been correctly set and I added a test case testProjectWithAliasFromScan(https://github.com/apache/calcite/pull/1351/files#diff-9f6a8f13d9ad70d7336340bf6b2e7b71R702) Thanks a lot for review this issue [~julianhyde][~rubenql] and sorry for my late reply > RelBuilder doesn't keep the alias when scan from an expanded view > - > > Key: CALCITE-3226 > URL: https://issues.apache.org/jira/browse/CALCITE-3226 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: jin xing >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In current implementation of {{RelBuilder::scan}} > ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]), > the alias can be derived and recorded into {{Frame}} only when the > {{RelNode}} is a {{TableScan}} > ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]). > But when {{RelBuilder::scan}} from an expanded view, the node is not a > {{TableScan}} and the alias is not kept. Below test failed - we cannot > reference a field by alias "MYVIEW". > {code:java} > @Test public void testExpandViewShouldKeepAlias() throws SQLException { > try (Connection connection = > DriverManager.getConnection("jdbc:calcite:")) { > final Frameworks.ConfigBuilder configBuilder = > expandingConfig(connection); > final RelOptTable.ViewExpander viewExpander = > (RelOptTable.ViewExpander) > Frameworks.getPlanner(configBuilder.build()); > final RelFactories.TableScanFactory tableScanFactory = > RelFactories.expandingScanFactory(viewExpander, > RelFactories.DEFAULT_TABLE_SCAN_FACTORY); > configBuilder.context(Contexts.of(tableScanFactory)); > final RelBuilder builder = RelBuilder.create(configBuilder.build()); > RelNode node = > builder.scan("MYVIEW") > .project( > builder.field(1, "MYVIEW", "EMPNO"), // Exception > thrown from here. > builder.field(1, "MYVIEW", "ENAME")) > .build(); > String expected = > "LogicalProject(EMPNO=[$0], ENAME=[$1])\n" > + " LogicalFilter(condition=[=(1, 1)])\n" > + "LogicalTableScan(table=[[scott, EMP]])\n"; > assertThat(node, hasTree(expected)); > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-3213) Support complex type expressions for SqlDataTypeSpec
[ https://issues.apache.org/jira/browse/CALCITE-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909552#comment-16909552 ] Danny Chan commented on CALCITE-3213: - Let me summarize the broad changes: * Support row type * Support array type * Support nested collection type * Support time zone definition for time and timestamp type * Support collection type for UDT and fix it's nullability > Support complex type expressions for SqlDataTypeSpec > - > > Key: CALCITE-3213 > URL: https://issues.apache.org/jira/browse/CALCITE-3213 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.21.0 > > > We should support nested struct type like: > {code:sql} > ROW( > foo NUMBER(5, 2) NOT NULL, > rec ROW(b BOOLEAN, i MyUDT NOT NULL)) > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (CALCITE-3255) CannotPlanException: There are not enough rules to produce a node with desired properties
[ https://issues.apache.org/jira/browse/CALCITE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chan resolved CALCITE-3255. - Resolution: Invalid Assignee: Danny Chan > CannotPlanException: There are not enough rules to produce a node with > desired properties > - > > Key: CALCITE-3255 > URL: https://issues.apache.org/jira/browse/CALCITE-3255 > Project: Calcite > Issue Type: Bug > Environment: !image-2019-08-16-18-09-57-964.png! >Reporter: Water Cut Off >Assignee: Danny Chan >Priority: Major > Attachments: image-2019-08-16-18-09-57-964.png > > > I want to use calcite volcano planner to estimate sql cost,but it doesn't > work when i try to test a case,The exception is blow: > {color:#FF}org.apache.calcite.plan.RelOptPlanner$CannotPlanException: > There are not enough rules to produce a node with desired properties: > convention=NONE, sort=[]. All the inputs have relevant nodes, however the > cost is still infinite.{color} > {color:#FF} Root: rel#40:Subset#4.NONE.[]{color} > {color:#FF} Original rel:{color} > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-3255) CannotPlanException: There are not enough rules to produce a node with desired properties
[ https://issues.apache.org/jira/browse/CALCITE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909539#comment-16909539 ] Danny Chan commented on CALCITE-3255: - Thanks, [~Water Cut Off], this is an expected behavior, The relational node cost would be infinite if the convention is NONE, you should first set up your desired conventions in the target target traits[1], and define you node transform converters[2]. You should also supply the metadata in you convention nodes, so that our metadata system can compute the cumulative cost correctly. [1] [https://github.com/apache/calcite/blob/a68bfc96bacc95b4b8ab5d185bb5cc1991113769/core/src/main/java/org/apache/calcite/tools/Program.java#L38] [2] https://github.com/apache/calcite/blob/a68bfc96bacc95b4b8ab5d185bb5cc1991113769/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L35 > CannotPlanException: There are not enough rules to produce a node with > desired properties > - > > Key: CALCITE-3255 > URL: https://issues.apache.org/jira/browse/CALCITE-3255 > Project: Calcite > Issue Type: Bug > Environment: !image-2019-08-16-18-09-57-964.png! >Reporter: Water Cut Off >Priority: Major > Attachments: image-2019-08-16-18-09-57-964.png > > > I want to use calcite volcano planner to estimate sql cost,but it doesn't > work when i try to test a case,The exception is blow: > {color:#FF}org.apache.calcite.plan.RelOptPlanner$CannotPlanException: > There are not enough rules to produce a node with desired properties: > convention=NONE, sort=[]. All the inputs have relevant nodes, however the > cost is still infinite.{color} > {color:#FF} Root: rel#40:Subset#4.NONE.[]{color} > {color:#FF} Original rel:{color} > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-3250) Support nested collection type for SqlDataTypeSpec
[ https://issues.apache.org/jira/browse/CALCITE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909535#comment-16909535 ] Danny Chan commented on CALCITE-3250: - Indeed, very sorry about that, i should have made the issue title and commit message more accurate. > Support nested collection type for SqlDataTypeSpec > --- > > Key: CALCITE-3250 > URL: https://issues.apache.org/jira/browse/CALCITE-3250 > Project: Calcite > Issue Type: Sub-task > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 20m > Remaining Estimate: 0h > > According to SQL-2011, collection type can be nested: > {code:sql} > ::= >ARRAY [bracket or trigraph> ] > ::= > > {code} > *Example*: > {code:sql} > int array array; > int multiset array; > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909486#comment-16909486 ] Xiening Dai commented on CALCITE-3257: -- I agree with your assumption. Current cost propagation logic wouldn't work if the parent node cost is already exists in the cache - in which case there's no chance for recomputing the cost based on the change of input cost. I think that's why we purge cache every time the rule fires (https://github.com/apache/calcite/blob/979a825fab0757c506363f5df6ed7997674e8726/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoRuleCall.java#L130). *RuleQueue.dump() calls into dumpGraphviz() which indeed calls getCost() for rel node*. At that point, new cache entry is added, causing problem in the next rule cost propagation. Your change might as well fix this particular case in the bug. But I think it's still important to keep the behavior consistent regardless the log level, aka purging the cache between rules. In the future, we might be able to improve the cache so that it can be reused across rules, which would be a huge gain of performance. But before that, I think we still need this fix so cache is purged after each dump(). > Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz > > > Key: CALCITE-3257 > URL: https://issues.apache.org/jira/browse/CALCITE-3257 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Xiening Dai >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > To repro - > 1. Set Log4J log level to TRACE. So planner will dump rel node info every > time at rule match > (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) > 2. Run JdbcTest.testNotExistsCorrelated. Get below exception - > java.lang.AssertionError: rel > [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has > lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 > cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]] > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) > at > org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) > at > org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) > at > org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) > at > org.apache.calcite.test.CalciteAssert$AssertQue
[jira] [Commented] (CALCITE-3254) AssertionError while deserializing of interval type.
[ https://issues.apache.org/jira/browse/CALCITE-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909481#comment-16909481 ] Wang Yanlin commented on CALCITE-3254: -- OK, I update the descritpion in pr. Thanks for reviewing. > AssertionError while deserializing of interval type. > > > Key: CALCITE-3254 > URL: https://issues.apache.org/jira/browse/CALCITE-3254 > Project: Calcite > Issue Type: Bug >Reporter: Wang Yanlin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When deserializing of logical rel haveing interval type, AssertionError > occurs. > The exception stacktrace as follow: > {code:java} > java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.AssertionError: use createSqlIntervalType() instead > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143) > at > org.apache.calcite.plan.RelWriterTest.testInterval(RelWriterTest.java:639) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) > Caused by: java.lang.RuntimeException: java.lang.AssertionError: use > createSqlIntervalType() instead > at > org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:271) > at > org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91) > at > org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85) > at > org.apache.calcite.plan.RelWriterTest.lambda$testInterval$8(RelWriterTest.java:644) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) > ... 25 more > Caused by: java.lang.AssertionError: use createSqlIntervalType() instead > at > org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:222) > at > org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:75) > at org.apache.calcite.rel.externalize.RelJson.toType(RelJson.java:222) > at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:516) > at > org.apache.calcite.rel.externalize.RelJson.toRexList(RelJson.java:607) > at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:447) > at > org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204) > at org.apache.calcite.rel.core.Project.(Project.java:100) > at > org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructor
[jira] [Commented] (CALCITE-3258) Upgrade jackson-databind from 2.9.9 to 2.9.9.2
[ https://issues.apache.org/jira/browse/CALCITE-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909460#comment-16909460 ] Julian Hyde commented on CALCITE-3258: -- I'll try 2.9.9.3. > Upgrade jackson-databind from 2.9.9 to 2.9.9.2 > -- > > Key: CALCITE-3258 > URL: https://issues.apache.org/jira/browse/CALCITE-3258 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Priority: Major > > Upgrade jackson-databind from 2.9.9 to 2.9.9.2. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909448#comment-16909448 ] Volodymyr Vysotskyi commented on CALCITE-3257: -- I don't know the exact reason for adding calls to invalidate rel nodes after the rule is applied, so I will operate only my assumptions. I think that it was done to force recalculating rel node cost after the rule is applied, so ideally we would have the correct cost for both unchanged and updated rel nodes - when the rule is applied, child rel nodes were changed and cost of current rel node was calculated considering children costs instead of using the outdated cached value. So ideally for the case when the tree wasn't changed, there is no need to invalidate the cache, since cached values should be correct. One of the issues, fixed in the scope of CALCITE-2018 was the case when several instances caches are used in the same time - one cache instance was invalidated after the rule is applied, but another one was used during the next rule applying. *As I see from the code, {{RuleQueue.dump()}} does not calculate rel nodes cost...* I think something similar happens in this case so invalidating the cache after (or perhaps also before) calling {{dump()}} method will help to avoid this issue, but the problem with coexisting several cache instances will still be present. Could you please check that CALCITE-2018 fixes the issue you have observed? > Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz > > > Key: CALCITE-3257 > URL: https://issues.apache.org/jira/browse/CALCITE-3257 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Xiening Dai >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > To repro - > 1. Set Log4J log level to TRACE. So planner will dump rel node info every > time at rule match > (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) > 2. Run JdbcTest.testNotExistsCorrelated. Get below exception - > java.lang.AssertionError: rel > [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has > lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 > cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]] > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) > at > org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) > at > org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) > at > org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) > at > org.ap
[jira] [Resolved] (CALCITE-3214) Add UnionToUnionRule for materialization matching
[ https://issues.apache.org/jira/browse/CALCITE-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3214. Resolution: Fixed Fix Version/s: 1.21.0 Fixed in https://github.com/apache/calcite/commit/a68bfc96bacc95b4b8ab5d185bb5cc1991113769, thanks for the PR, [~jinxing6...@126.com]! > Add UnionToUnionRule for materialization matching > - > > Key: CALCITE-3214 > URL: https://issues.apache.org/jira/browse/CALCITE-3214 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: jin xing >Assignee: jin xing >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Below materialization matching fails now > {code:java} > @Test public void testDEV() { > String sql0 = "select * from \"emps\" where \"empid\" < 300"; > String sql1 = "select * from \"emps\" where \"empid\" > 200"; > checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + > sql0); > } > {code} > This issue proposes to add a rule for union matching -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (CALCITE-3249) Substitution#getRexShuttle does not consider RexLiteral
[ https://issues.apache.org/jira/browse/CALCITE-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3249. Resolution: Fixed Fix Version/s: 1.21.0 Fixed in https://github.com/apache/calcite/commit/0a6f53cb828cd39d78d1423361921bf40636943a, thanks for the PR, [~jinxing6...@126.com]! > Substitution#getRexShuttle does not consider RexLiteral > --- > > Key: CALCITE-3249 > URL: https://issues.apache.org/jira/browse/CALCITE-3249 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: jin xing >Assignee: jin xing >Priority: Minor > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Current {{Substitution#getRexShuttle}} does not take {{RexLiteral}} into > consideration. > Thus below query & mv fails matching: > {code:java} > MV: > select deptno, sum(salary), sum(commission), sum(k) > from > (select deptno, salary, commission, 100 as k > from emps) > group by deptno > Query: > select deptno, sum(salary), sum(k) > from > (select deptno, salary, 100 as k > from emps) > group by deptno > {code} > The root cause is that {{ProjectToProjectUnifyRule}} compensates a > {{Project}} which contains {{RexLiteral}}, but > {{AggregateOnProjectToAggregateUnifyRule}} works only when the {{Project}} in > query is a mapping > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1357 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-2018) Queries failed with AssertionError: rel has lower cost than best cost of subset
[ https://issues.apache.org/jira/browse/CALCITE-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909415#comment-16909415 ] Xiening Dai commented on CALCITE-2018: -- Hi [~julianhyde], [~vvysotskyi] also has some questions. We can continue the discussion under CALCITE-3257. Once we agree on the problem, I would update/clarify the CL. Thanks. > Queries failed with AssertionError: rel has lower cost than best cost of > subset > --- > > Key: CALCITE-2018 > URL: https://issues.apache.org/jira/browse/CALCITE-2018 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.13.0 >Reporter: Volodymyr Vysotskyi >Assignee: Danny Chan >Priority: Critical > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > *Problem description* > When rootLogger level is DEBUG, unit tests > * MaterializationTest.testMaterializationSubstitution2 > * MaterializationTest.testJoinMaterializationUKFK8 > * MaterializationTest.testJoinMaterializationUKFK6 > * JdbcTest.testWhereNot > unit tests are failed with error AssertionError: rel has lower cost than best > cost of subset. > Full stack trace for test > {{MaterializationTest.testMaterializationSubstitution2}}: > {noformat} > java.lang.AssertionError: rel > [rel#245:EnumerableUnion.ENUMERABLE.[](input#0=rel#246:Subset#5.ENUMERABLE.[],input#1=rel#239:Subset#6.ENUMERABLE.[0],all=true)] > has lower cost {14.0 rows, 19.0 cpu, 0.0 io} than best cost {15.0 rows, 20.0 > cpu, 0.0 io} of subset [rel#243:Subset#7.ENUMERABLE.[]] > at > org.apache.calcite.plan.volcano.VolcanoPlanner.validate(VolcanoPlanner.java:906) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:866) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:101) > at > org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:336) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1495) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:863) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1766) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234) > at > org.apache.calcite.rel.rules.FilterProjectTransposeRule.onMatch(FilterProjectTransposeRule.java:143) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650) > at org.apache.calcite.tools.Programs$5.run(Programs.java:326) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:387) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:187) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:318) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:229) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:786) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:640) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:610) > at org.apache.calcite.schema.Schemas.prepare(Schemas.java:346) > at > org.apache.calcite.materialize.MaterializationService$DefaultTableFactory.createTable(MaterializationService.java:374) > at > org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:137) > at > org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:99) > at > org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:110) > at > org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:100) > at > org.apache.calcite.schema.impl.MaterializedViewTable.create(MaterializedViewTable.java:81) > at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:364) > at > org.apache.calcite.model.JsonMaterialization.accept(JsonMaterialization.java:42) > at org.apache.calcite.model.JsonSchema.visitChildren(JsonSchema.java:98) > at > org.apache.calcite.model.JsonMapSchema.visitChildren(J
[jira] [Commented] (CALCITE-3258) Upgrade jackson-databind from 2.9.9 to 2.9.9.2
[ https://issues.apache.org/jira/browse/CALCITE-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909408#comment-16909408 ] Vladimir Sitnikov commented on CALCITE-3258: I would suggest going with 2.9.9.3. Is there a reason for not using it? > Upgrade jackson-databind from 2.9.9 to 2.9.9.2 > -- > > Key: CALCITE-3258 > URL: https://issues.apache.org/jira/browse/CALCITE-3258 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Priority: Major > > Upgrade jackson-databind from 2.9.9 to 2.9.9.2. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909402#comment-16909402 ] Xiening Dai commented on CALCITE-3257: -- I believe CALCITE-2018 is a separate issue while this one needs to be fixed too. From my understanding, the meta cache needs to be purged for every rule firing, dump() breaks this assumption. Adding this call would make sure cache is purged and would actually fix quite a number of failures under TRACE mode. But there are still cases where best cost is not properly calculated (CALCITE-2018). This fix cannot address those cases (one example JdbcTest.testInnerJoinValues) thus it doesn't "hide" anything. > Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz > > > Key: CALCITE-3257 > URL: https://issues.apache.org/jira/browse/CALCITE-3257 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Xiening Dai >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > To repro - > 1. Set Log4J log level to TRACE. So planner will dump rel node info every > time at rule match > (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) > 2. Run JdbcTest.testNotExistsCorrelated. Get below exception - > java.lang.AssertionError: rel > [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has > lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 > cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]] > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) > at > org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) > at > org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) > at > org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556) > at > org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method
[jira] [Created] (CALCITE-3258) Upgrade jackson-databind from 2.9.9 to 2.9.9.2
Julian Hyde created CALCITE-3258: Summary: Upgrade jackson-databind from 2.9.9 to 2.9.9.2 Key: CALCITE-3258 URL: https://issues.apache.org/jira/browse/CALCITE-3258 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Upgrade jackson-databind from 2.9.9 to 2.9.9.2. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909375#comment-16909375 ] Volodymyr Vysotskyi commented on CALCITE-3257: -- I don't think that adding a call to RelOptCluster.invalidateMetadataQuery() after dumping rel nodes is a good idea. As pointed in one of the comments in CALCITE-2018, this issue also happens for the case when a trace is not enabled. The proposed fix will just hide the issue for some cases, but more important and hard-to-investigate issues like incorrect planning and others will still be present. > Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz > > > Key: CALCITE-3257 > URL: https://issues.apache.org/jira/browse/CALCITE-3257 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Xiening Dai >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > To repro - > 1. Set Log4J log level to TRACE. So planner will dump rel node info every > time at rule match > (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) > 2. Run JdbcTest.testNotExistsCorrelated. Get below exception - > java.lang.AssertionError: rel > [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has > lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 > cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]] > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) > at > org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) > at > org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) > at > org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556) > at > org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflecti
[jira] [Updated] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-3257: Labels: pull-request-available (was: ) > Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz > > > Key: CALCITE-3257 > URL: https://issues.apache.org/jira/browse/CALCITE-3257 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Xiening Dai >Priority: Major > Labels: pull-request-available > > To repro - > 1. Set Log4J log level to TRACE. So planner will dump rel node info every > time at rule match > (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) > 2. Run JdbcTest.testNotExistsCorrelated. Get below exception - > java.lang.AssertionError: rel > [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has > lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 > cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]] > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) > at > org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) > at > org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) > at > org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556) > at > org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunne
[jira] [Updated] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiening Dai updated CALCITE-3257: - Description: To repro - 1. Set Log4J log level to TRACE. So planner will dump rel node info every time at rule match (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) 2. Run JdbcTest.testNotExistsCorrelated. Get below exception - java.lang.AssertionError: rel [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]] at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) at org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889) at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928) at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) at org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) at org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) at org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) at org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556) at org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runn
[jira] [Commented] (CALCITE-3250) Support nested collection type for SqlDataTypeSpec
[ https://issues.apache.org/jira/browse/CALCITE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909307#comment-16909307 ] Julian Hyde commented on CALCITE-3250: -- Also, this is an extension to SQL grammar, right? Lead with that. You make it sound like it was just a change to the SqlDataTypeSpec API. > Support nested collection type for SqlDataTypeSpec > --- > > Key: CALCITE-3250 > URL: https://issues.apache.org/jira/browse/CALCITE-3250 > Project: Calcite > Issue Type: Sub-task > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 20m > Remaining Estimate: 0h > > According to SQL-2011, collection type can be nested: > {code:sql} > ::= >ARRAY [bracket or trigraph> ] > ::= > > {code} > *Example*: > {code:sql} > int array array; > int multiset array; > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (CALCITE-3246) RelJsonReader throws NullPointerException while deserializing from JSON a call to a user-defined function (UDF)
[ https://issues.apache.org/jira/browse/CALCITE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated CALCITE-3246: - Summary: RelJsonReader throws NullPointerException while deserializing from JSON a call to a user-defined function (UDF) (was: NullPointerException while deserializing of udf.) > RelJsonReader throws NullPointerException while deserializing from JSON a > call to a user-defined function (UDF) > --- > > Key: CALCITE-3246 > URL: https://issues.apache.org/jira/browse/CALCITE-3246 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Wang Yanlin >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > when deserializing of logical rel with udf operator, NPE occurs. > The exception stacktrace as follow. > {code:java} > java.lang.RuntimeException: java.lang.NullPointerException > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143) > at org.apache.calcite.plan.RelWriterTest.testUdf(RelWriterTest.java:598) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) > Caused by: java.lang.NullPointerException > at java.util.Objects.requireNonNull(Objects.java:203) > at org.apache.calcite.rex.RexCall.(RexCall.java:83) > at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:237) > at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:485) > at > org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204) > at org.apache.calcite.rel.core.Project.(Project.java:100) > at > org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:261) > at > org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91) > at > org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85) > at > org.apache.calcite.plan.RelWriterTest.lambda$testUdf$7(RelWriterTest.java:603) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Fr
[jira] [Commented] (CALCITE-3246) NullPointerException while deserializing of udf.
[ https://issues.apache.org/jira/browse/CALCITE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909287#comment-16909287 ] Julian Hyde commented on CALCITE-3246: -- I force-pushed to master to remove your commit. Let's commit when we're ready. > NullPointerException while deserializing of udf. > - > > Key: CALCITE-3246 > URL: https://issues.apache.org/jira/browse/CALCITE-3246 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Wang Yanlin >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > when deserializing of logical rel with udf operator, NPE occurs. > The exception stacktrace as follow. > {code:java} > java.lang.RuntimeException: java.lang.NullPointerException > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143) > at org.apache.calcite.plan.RelWriterTest.testUdf(RelWriterTest.java:598) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) > Caused by: java.lang.NullPointerException > at java.util.Objects.requireNonNull(Objects.java:203) > at org.apache.calcite.rex.RexCall.(RexCall.java:83) > at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:237) > at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:485) > at > org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204) > at org.apache.calcite.rel.core.Project.(Project.java:100) > at > org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:261) > at > org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91) > at > org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85) > at > org.apache.calcite.plan.RelWriterTest.lambda$testUdf$7(RelWriterTest.java:603) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) > ... 25 more > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-2018) Queries failed with AssertionError: rel has lower cost than best cost of subset
[ https://issues.apache.org/jira/browse/CALCITE-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909209#comment-16909209 ] Julian Hyde commented on CALCITE-2018: -- I saw the issue just logged for this workaround. But it doesn’t make clear that it is just a workaround, nor state its relationship to this, the issue that describes the root cause. Can you make sure that the other issue is in proper context. > Queries failed with AssertionError: rel has lower cost than best cost of > subset > --- > > Key: CALCITE-2018 > URL: https://issues.apache.org/jira/browse/CALCITE-2018 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.13.0 >Reporter: Volodymyr Vysotskyi >Assignee: Danny Chan >Priority: Critical > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > *Problem description* > When rootLogger level is DEBUG, unit tests > * MaterializationTest.testMaterializationSubstitution2 > * MaterializationTest.testJoinMaterializationUKFK8 > * MaterializationTest.testJoinMaterializationUKFK6 > * JdbcTest.testWhereNot > unit tests are failed with error AssertionError: rel has lower cost than best > cost of subset. > Full stack trace for test > {{MaterializationTest.testMaterializationSubstitution2}}: > {noformat} > java.lang.AssertionError: rel > [rel#245:EnumerableUnion.ENUMERABLE.[](input#0=rel#246:Subset#5.ENUMERABLE.[],input#1=rel#239:Subset#6.ENUMERABLE.[0],all=true)] > has lower cost {14.0 rows, 19.0 cpu, 0.0 io} than best cost {15.0 rows, 20.0 > cpu, 0.0 io} of subset [rel#243:Subset#7.ENUMERABLE.[]] > at > org.apache.calcite.plan.volcano.VolcanoPlanner.validate(VolcanoPlanner.java:906) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:866) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:101) > at > org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:336) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1495) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:863) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1766) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234) > at > org.apache.calcite.rel.rules.FilterProjectTransposeRule.onMatch(FilterProjectTransposeRule.java:143) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650) > at org.apache.calcite.tools.Programs$5.run(Programs.java:326) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:387) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:187) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:318) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:229) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:786) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:640) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:610) > at org.apache.calcite.schema.Schemas.prepare(Schemas.java:346) > at > org.apache.calcite.materialize.MaterializationService$DefaultTableFactory.createTable(MaterializationService.java:374) > at > org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:137) > at > org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:99) > at > org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:110) > at > org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:100) > at > org.apache.calcite.schema.impl.MaterializedViewTable.create(MaterializedViewTable.java:81) > at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:364) > at > org.apache.calcite.model.JsonMaterialization.accept(JsonMaterialization.java:42) > at org.apache.calcite.model.JsonSchema.visitChildren(JsonSchema.java:98)
[jira] [Commented] (CALCITE-3254) AssertionError while deserializing of interval type.
[ https://issues.apache.org/jira/browse/CALCITE-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909202#comment-16909202 ] Julian Hyde commented on CALCITE-3254: -- Can you improve the description? Think about whether you are fixing a bug or adding a feature. > AssertionError while deserializing of interval type. > > > Key: CALCITE-3254 > URL: https://issues.apache.org/jira/browse/CALCITE-3254 > Project: Calcite > Issue Type: Bug >Reporter: Wang Yanlin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When deserializing of logical rel haveing interval type, AssertionError > occurs. > The exception stacktrace as follow: > {code:java} > java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.AssertionError: use createSqlIntervalType() instead > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143) > at > org.apache.calcite.plan.RelWriterTest.testInterval(RelWriterTest.java:639) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) > Caused by: java.lang.RuntimeException: java.lang.AssertionError: use > createSqlIntervalType() instead > at > org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:271) > at > org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91) > at > org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85) > at > org.apache.calcite.plan.RelWriterTest.lambda$testInterval$8(RelWriterTest.java:644) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) > ... 25 more > Caused by: java.lang.AssertionError: use createSqlIntervalType() instead > at > org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:222) > at > org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:75) > at org.apache.calcite.rel.externalize.RelJson.toType(RelJson.java:222) > at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:516) > at > org.apache.calcite.rel.externalize.RelJson.toRexList(RelJson.java:607) > at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:447) > at > org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204) > at org.apache.calcite.rel.core.Project.(Project.java:100) > at > org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >
[jira] [Updated] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiening Dai updated CALCITE-3257: - Description: To repro - 1. Set Log4J log level to TRACE. So planner will dump rel node info every time at rule match (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) 2. Run JdbcTest.testInnerJoinValues. Get below exception - java.lang.AssertionError: rel [rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)] has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, 241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]] at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) at org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888) at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90) at org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329) at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656) at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927) at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) at org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) at org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) at org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) at org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556) at org.apache.calcite.test.JdbcTest.testInnerJoinValues(JdbcTest.java:2580) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runC
[jira] [Comment Edited] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909156#comment-16909156 ] Xiening Dai edited comment on CALCITE-3257 at 8/16/19 4:05 PM: --- Based on current design, RelMetadataQuery.map needs to be cleared between each rule firing. This is achieved through RelOptCluster.invalidateMetadataQuery() by VolcanoRuleCall.transformTo(). But when trace is enabled, the dump process would actually rebuild the meta data cache from previous rel tree. Then the subsequent rule firing doesn't get a chance to update rel node cost as it's been in the cache. A simple fix will just add a call to RelOptCluster.invalidateMetadataQuery() after dumping rel nodes. was (Author: xndai): Based on current design, RelMetadataQuery.map needs to be cleared between each rule firing. This is achieve through RelOptCluster.invalidateMetadataQuery() by VolcanoRuleCall.transformTo(). But when trace is enabled, the dump process would actually rebuild the meta data cache from previous rel tree. Then the subsequent rule firing doesn't get a chance to update rel node cost as it's been in the cache. A simple fix will just add a call to RelOptCluster.invalidateMetadataQuery() after dumping rel nodes. > Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz > > > Key: CALCITE-3257 > URL: https://issues.apache.org/jira/browse/CALCITE-3257 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Xiening Dai >Priority: Major > > To repro - > 1. Set Log4J log level to TRACE. So planner will dump rel node info every > time at rule match > (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) > 2. Run JdbcTest.testInnerJoinValues. Get below exception - > java.lang.AssertionError: rel > [rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)] > has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, > 241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]] > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90) > at > org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) > at > org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) > at > org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatem
[jira] [Updated] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiening Dai updated CALCITE-3257: - Description: To repro - 1. Set Log4J log level to TRACE. So planner will dump rel node info every time at rule match (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) 2. Run JdbcTest.testInnerJoinValues. Get below exception - java.lang.AssertionError: rel [rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)] has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, 241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]] at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) at org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888) at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90) at org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329) at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656) at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927) at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) at org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) at org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) at org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) at org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556) at org.apache.calcite.test.JdbcTest.testInnerJoinValues(JdbcTest.java:2580) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runC
[jira] [Created] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
Xiening Dai created CALCITE-3257: Summary: Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz Key: CALCITE-3257 URL: https://issues.apache.org/jira/browse/CALCITE-3257 Project: Calcite Issue Type: Bug Components: core Reporter: Xiening Dai To repro - 1. Set Log4J log level to TRACE. So planner will dump rel node info every time at rule match (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) 2. Run JdbcTest.testInnerJoinValues. Get below exception - java.lang.AssertionError: rel [rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)] has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, 241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]] at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) at org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888) at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90) at org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329) at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656) at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927) at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) at org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) at org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) at org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) at org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556) at org.apache.calcite.test.JdbcTest.testInnerJoinValues(JdbcTest.java:2580) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runL
[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[ https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909156#comment-16909156 ] Xiening Dai commented on CALCITE-3257: -- Based on current design, RelMetadataQuery.map needs to be cleared between each rule firing. This is achieve through RelOptCluster.invalidateMetadataQuery() by VolcanoRuleCall.transformTo(). But when trace is enabled, the dump process would actually rebuild the meta data cache from previous rel tree. Then the subsequent rule firing doesn't get a chance to update rel node cost as it's been in the cache. A simple fix will just add a call to RelOptCluster.invalidateMetadataQuery() after dumping rel nodes. > Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz > > > Key: CALCITE-3257 > URL: https://issues.apache.org/jira/browse/CALCITE-3257 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Xiening Dai >Priority: Major > > To repro - > 1. Set Log4J log level to TRACE. So planner will dump rel node info every > time at rule match > (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435) > 2. Run JdbcTest.testInnerJoinValues. Get below exception - > java.lang.AssertionError: rel > [rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)] > has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, > 241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]] > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90) > at > org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129) > at > org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236) > at > org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631) > at > org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286) > at > org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346) > at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502) > at > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231) > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) > at > org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561) > at > org.apache.calcite.test.CalciteAssert$AssertQuery.explainCo
[jira] [Commented] (CALCITE-3246) NullPointerException while deserializing of udf.
[ https://issues.apache.org/jira/browse/CALCITE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909016#comment-16909016 ] Wang Yanlin commented on CALCITE-3246: -- The class name is put into the json string in [to json function|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/externalize/RelJson.java#L366]. Any more details on how to check the security implications? > NullPointerException while deserializing of udf. > - > > Key: CALCITE-3246 > URL: https://issues.apache.org/jira/browse/CALCITE-3246 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Wang Yanlin >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > when deserializing of logical rel with udf operator, NPE occurs. > The exception stacktrace as follow. > {code:java} > java.lang.RuntimeException: java.lang.NullPointerException > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143) > at org.apache.calcite.plan.RelWriterTest.testUdf(RelWriterTest.java:598) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) > Caused by: java.lang.NullPointerException > at java.util.Objects.requireNonNull(Objects.java:203) > at org.apache.calcite.rex.RexCall.(RexCall.java:83) > at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:237) > at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:485) > at > org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204) > at org.apache.calcite.rel.core.Project.(Project.java:100) > at > org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:261) > at > org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91) > at > org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85) > at > org.apache.calcite.plan.RelWriterTest.lambda$testUdf$7(RelWriterTest.java:603) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks
[jira] [Updated] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.
[ https://issues.apache.org/jira/browse/CALCITE-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-3256: Labels: pull-request-available (was: ) > Add ProjectOnProjectToProjectUnifyRule for materialization matching. > > > Key: CALCITE-3256 > URL: https://issues.apache.org/jira/browse/CALCITE-3256 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: jin xing >Assignee: jin xing >Priority: Minor > Labels: pull-request-available > > In current implementation of rules in SubstitutionVisitor.java & > MaterializedViewSubstitutionVisitor.java, it's quite common to add a > compensating Project on top of child node of target(MV-rel) during matching. > But afterwards the next round matching should be able to handle such a > compensated Project and match upward along the plan tree. Otherwise we fail > the matching. After all, the goal of matching is to transform the query and > let a complete 'target' show up in the transformed query plan. > I found cases where the compensated Project cannot be properly handled. > {code:java} > MV: > select deptno, sum(salary) + 2, sum(commission) > from emps > group by deptno > Query: > select deptno, sum(salary) + 2 > from emps > group by deptno > {code} > After matching of the Aggregates, a compensating Project is added, but > afterwards matching fails to handle it. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.
jin xing created CALCITE-3256: - Summary: Add ProjectOnProjectToProjectUnifyRule for materialization matching. Key: CALCITE-3256 URL: https://issues.apache.org/jira/browse/CALCITE-3256 Project: Calcite Issue Type: Improvement Components: core Reporter: jin xing Assignee: jin xing In current implementation of rules in SubstitutionVisitor.java & MaterializedViewSubstitutionVisitor.java, it's quite common to add a compensating Project on top of child node of target(MV-rel) during matching. But afterwards the next round matching should be able to handle such a compensated Project and match upward along the plan tree. Otherwise we fail the matching. After all, the goal of matching is to transform the query and let a complete 'target' show up in the transformed query plan. I found cases where the compensated Project cannot be properly handled. {code:java} MV: select deptno, sum(salary) + 2, sum(commission) from emps group by deptno Query: select deptno, sum(salary) + 2 from emps group by deptno {code} After matching of the Aggregates, a compensating Project is added, but afterwards matching fails to handle it. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (CALCITE-3255) CannotPlanException: There are not enough rules to produce a node with desired properties
[ https://issues.apache.org/jira/browse/CALCITE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Water Cut Off updated CALCITE-3255: --- Description: I want to use calcite volcano planner to estimate sql cost,but it doesn't work when i try to test a case,The exception is blow: {color:#FF}org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough rules to produce a node with desired properties: convention=NONE, sort=[]. All the inputs have relevant nodes, however the cost is still infinite.{color} {color:#FF} Root: rel#40:Subset#4.NONE.[]{color} {color:#FF} Original rel:{color} was: I want to use calcite volcano planner to estimate sql cost,but it doesn't work when i try to test a case,The exception is blow: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough rules to produce a node with desired properties: convention=NONE, sort=[]. All the inputs have relevant nodes, however the cost is still infinite. Root: rel#40:Subset#4.NONE.[] Original rel: > CannotPlanException: There are not enough rules to produce a node with > desired properties > - > > Key: CALCITE-3255 > URL: https://issues.apache.org/jira/browse/CALCITE-3255 > Project: Calcite > Issue Type: Bug > Environment: !image-2019-08-16-18-09-57-964.png! >Reporter: Water Cut Off >Priority: Major > Attachments: image-2019-08-16-18-09-57-964.png > > > I want to use calcite volcano planner to estimate sql cost,but it doesn't > work when i try to test a case,The exception is blow: > {color:#FF}org.apache.calcite.plan.RelOptPlanner$CannotPlanException: > There are not enough rules to produce a node with desired properties: > convention=NONE, sort=[]. All the inputs have relevant nodes, however the > cost is still infinite.{color} > {color:#FF} Root: rel#40:Subset#4.NONE.[]{color} > {color:#FF} Original rel:{color} > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (CALCITE-3255) CannotPlanException: There are not enough rules to produce a node with desired properties
Water Cut Off created CALCITE-3255: -- Summary: CannotPlanException: There are not enough rules to produce a node with desired properties Key: CALCITE-3255 URL: https://issues.apache.org/jira/browse/CALCITE-3255 Project: Calcite Issue Type: Bug Environment: !image-2019-08-16-18-09-57-964.png! Reporter: Water Cut Off Attachments: image-2019-08-16-18-09-57-964.png I want to use calcite volcano planner to estimate sql cost,but it doesn't work when i try to test a case,The exception is blow: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough rules to produce a node with desired properties: convention=NONE, sort=[]. All the inputs have relevant nodes, however the cost is still infinite. Root: rel#40:Subset#4.NONE.[] Original rel: -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (CALCITE-3254) AssertionError while deserializing of interval type.
[ https://issues.apache.org/jira/browse/CALCITE-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-3254: Labels: pull-request-available (was: ) > AssertionError while deserializing of interval type. > > > Key: CALCITE-3254 > URL: https://issues.apache.org/jira/browse/CALCITE-3254 > Project: Calcite > Issue Type: Bug >Reporter: Wang Yanlin >Priority: Major > Labels: pull-request-available > > When deserializing of logical rel haveing interval type, AssertionError > occurs. > The exception stacktrace as follow: > {code:java} > java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.AssertionError: use createSqlIntervalType() instead > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143) > at > org.apache.calcite.plan.RelWriterTest.testInterval(RelWriterTest.java:639) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) > Caused by: java.lang.RuntimeException: java.lang.AssertionError: use > createSqlIntervalType() instead > at > org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:271) > at > org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91) > at > org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85) > at > org.apache.calcite.plan.RelWriterTest.lambda$testInterval$8(RelWriterTest.java:644) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) > ... 25 more > Caused by: java.lang.AssertionError: use createSqlIntervalType() instead > at > org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:222) > at > org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:75) > at org.apache.calcite.rel.externalize.RelJson.toType(RelJson.java:222) > at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:516) > at > org.apache.calcite.rel.externalize.RelJson.toRexList(RelJson.java:607) > at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:447) > at > org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204) > at org.apache.calcite.rel.core.Project.(Project.java:100) > at > org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessor
[jira] [Created] (CALCITE-3254) AssertionError while deserializing of interval type.
Wang Yanlin created CALCITE-3254: Summary: AssertionError while deserializing of interval type. Key: CALCITE-3254 URL: https://issues.apache.org/jira/browse/CALCITE-3254 Project: Calcite Issue Type: Bug Reporter: Wang Yanlin When deserializing of logical rel haveing interval type, AssertionError occurs. The exception stacktrace as follow: {code:java} java.lang.RuntimeException: java.lang.RuntimeException: java.lang.AssertionError: use createSqlIntervalType() instead at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181) at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143) at org.apache.calcite.plan.RelWriterTest.testInterval(RelWriterTest.java:639) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Caused by: java.lang.RuntimeException: java.lang.AssertionError: use createSqlIntervalType() instead at org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:271) at org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91) at org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85) at org.apache.calcite.plan.RelWriterTest.lambda$testInterval$8(RelWriterTest.java:644) at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) ... 25 more Caused by: java.lang.AssertionError: use createSqlIntervalType() instead at org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:222) at org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:75) at org.apache.calcite.rel.externalize.RelJson.toType(RelJson.java:222) at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:516) at org.apache.calcite.rel.externalize.RelJson.toRexList(RelJson.java:607) at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:447) at org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204) at org.apache.calcite.rel.core.Project.(Project.java:100) at org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:261) ... 31 more {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (CALCITE-2302) Implicit type cast support
[ https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908903#comment-16908903 ] Danny Chan edited comment on CALCITE-2302 at 8/16/19 9:45 AM: -- Thanks so much for your review [~hyuan] ! I have add a new SqlConformance for PostgreSql and SqlServer which seem the only engine that returns integer for 2 integers division. I checked again for the behavior: * Hive and Spark all return double, Hive also have a "A DIV B" function to return integer * PostgreSql and SQL_SERVER returns integer. * Mysql returns double while it has a DIV() function to return integer was (Author: danny0405): Thanks so much for your review [~hyuan] ! I have add a new SqlConformance for PostgreSql and SqlServer which seem the only engine that returns integer for 2 integers division. I checked again for the behavior: * Hive and Spark all return double, Hive also have a "A DIV B" function to returns integer * PostgreSql and SQL_SERVER returns integer. * Mysql returns double while it has a DIV() function to returns integer > Implicit type cast support > -- > > Key: CALCITE-2302 > URL: https://issues.apache.org/jira/browse/CALCITE-2302 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.17.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive. > Implicit type cast is an useful function for many cases, So we should support > this. > I checkout Calcite code and found that: > # Now we use a validator to validate our operands types[ through kinds of > namespaces and scopes ] > # Most of the validations will finally goes to > {code:java} > SqlOperator.validateOperands > {code} > # which will use validation logic defined in corresponding > SqlOperandTypeChecker > What i'm confused about is where should i put the implicit type cast logic > in? I figured out 2 ways: > # Supply a tool class/rules to add casts into a parsed SqlNode tree which > will then go through the validation logic later on. > # Unleash the validation logic in kinds of SqlOperandTypeChecker, then > modify the RelNode/RexNodes tree converted from a validated SqlNode tree to > add in casts through custom RelOptRules. > So guys, which of the 2 ways should i go, or if there are better way to do > this? > I need your help. > > Updated 18-05-30: > Hi guys, i have made a PR in > [CALCITE-2302|https://github.com/apache/calcite/pull/706] > This is design doc: [Calcite Implicit Type Cast > Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing]. > This is the conversion types mapping: [Conversion Types > Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing]. > I really appreciate your suggestions, thx. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-2302) Implicit type cast support
[ https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908903#comment-16908903 ] Danny Chan commented on CALCITE-2302: - Thanks so much for your review [~hyuan] ! I have add a new SqlConformance for PostgreSql and SqlServer which seem the only engine that returns integer for 2 integers division. I checked again for the behavior: * Hive and Spark all return double, Hive also have a "A DIV B" function to returns integer * PostgreSql and SQL_SERVER returns integer. * Mysql returns double while it has a DIV() function to returns integer > Implicit type cast support > -- > > Key: CALCITE-2302 > URL: https://issues.apache.org/jira/browse/CALCITE-2302 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.17.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive. > Implicit type cast is an useful function for many cases, So we should support > this. > I checkout Calcite code and found that: > # Now we use a validator to validate our operands types[ through kinds of > namespaces and scopes ] > # Most of the validations will finally goes to > {code:java} > SqlOperator.validateOperands > {code} > # which will use validation logic defined in corresponding > SqlOperandTypeChecker > What i'm confused about is where should i put the implicit type cast logic > in? I figured out 2 ways: > # Supply a tool class/rules to add casts into a parsed SqlNode tree which > will then go through the validation logic later on. > # Unleash the validation logic in kinds of SqlOperandTypeChecker, then > modify the RelNode/RexNodes tree converted from a validated SqlNode tree to > add in casts through custom RelOptRules. > So guys, which of the 2 ways should i go, or if there are better way to do > this? > I need your help. > > Updated 18-05-30: > Hi guys, i have made a PR in > [CALCITE-2302|https://github.com/apache/calcite/pull/706] > This is design doc: [Calcite Implicit Type Cast > Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing]. > This is the conversion types mapping: [Conversion Types > Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing]. > I really appreciate your suggestions, thx. -- This message was sent by Atlassian JIRA (v7.6.14#76016)