[jira] [Resolved] (CALCITE-2776) Wrong value when accessing struct types with one attribute
[ https://issues.apache.org/jira/browse/CALCITE-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved CALCITE-2776. -- Resolution: Fixed Fixed in [5a305f166cf06419a0c28e496a9bf853ba540047|https://github.com/apache/calcite-avatica/commit/5a305f166cf06419a0c28e496a9bf853ba540047]. > Wrong value when accessing struct types with one attribute > -- > > Key: CALCITE-2776 > URL: https://issues.apache.org/jira/browse/CALCITE-2776 > Project: Calcite > Issue Type: Bug > Components: avatica >Affects Versions: avatica-1.13.0 >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Blocker > Labels: pull-request-available > Fix For: avatica-1.14.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The accessors for struct types with one attribute are not created correctly > leading to wrong values when the internal representation for structured types > is an Object[]. > For example consider the following query: > {code:sql} > select * from (values > (1, ROW(1)), > (2, ROW(2))) as v(id,struct); > {code} > Executing this query on calcite returns: > {code:sql} > +++ > | ID | STRUCT | > +++ > | 1 | {[Ljava.lang.Object;@4ff4357f} | > | 2 | {[Ljava.lang.Object;@49cb9cb5} | > +++ > {code} > instead of: > {code:sql} > +++ > | ID | STRUCT | > +++ > | 1 | {1}| > | 2 | {2}| > +++ > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-2866) Allow passing factory of SqlValidator / SqlToRelConverter into FrameworkConfig
[ https://issues.apache.org/jira/browse/CALCITE-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongze Zhang updated CALCITE-2866: -- Description: It seems that the customization of SqlValidator / SqlToRelConverter / CatalogReader in downstream projects is normal. E.g. Following are some examples (not all) of extended SqlValidator / SqlToRelConverter / CatalogReader from other projects that are using Calcite as query planner : # DrillValidator [https://github.com/apache/drill/blob/4627973bde9847a4eb2672c44941136c167326a1/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java#L249] # FlinkCalciteSqlValidator [https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/calcite/FlinkCalciteSqlValidator.scala#L31] # DremioSqlToRelConverter [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/planner/sql/DremioSqlToRelConverter.java#L40] # (Dremio) SqlValidatorImpl [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/planner/sql/SqlValidatorImpl.java#L32] # DremioCatalogReader [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/catalog/DremioCatalogReader.java#L71] The class tools/Planner.java[1] is a good customization tool for Calcite user. However currently there are no way to use Frameworks.getPlanner() directly if the classes need to be customized[2][3]. [1] [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/Planner.java] [2] [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L232] [3] [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L184] was: It seems that the customization of SqlValidator / SqlToRelConverter in downstream projects is normal. E.g. Following are some examples (not all) of extended SqlValidator or SqlToRelConverter from other projects that are using Calcite as query planner : # DrillValidator [https://github.com/apache/drill/blob/4627973bde9847a4eb2672c44941136c167326a1/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java#L249] # FlinkCalciteSqlValidator [https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/calcite/FlinkCalciteSqlValidator.scala#L31] # DremioSqlToRelConverter [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/planner/sql/DremioSqlToRelConverter.java#L40] # (Dremio) SqlValidatorImpl [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/planner/sql/SqlValidatorImpl.java#L32] The class tools/Planner.java[1] is a good customization tool for Calcite user. However currently there are no way to use Frameworks.getPlanner() directly if the two class need to be customized[2][3]. [1] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/Planner.java [2] [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L232] [3] [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L184] > Allow passing factory of SqlValidator / SqlToRelConverter into FrameworkConfig > -- > > Key: CALCITE-2866 > URL: https://issues.apache.org/jira/browse/CALCITE-2866 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Hongze Zhang >Assignee: Hongze Zhang >Priority: Major > Labels: usability > > It seems that the customization of SqlValidator / SqlToRelConverter / > CatalogReader in downstream projects is normal. > E.g. Following are some examples (not all) of extended SqlValidator / > SqlToRelConverter / CatalogReader from other projects that are using Calcite > as query planner : > # DrillValidator > > [https://github.com/apache/drill/blob/4627973bde9847a4eb2672c44941136c167326a1/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java#L249] > # FlinkCalciteSqlValidator > > [https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/calcite/FlinkCalciteSqlValidator.scala#L31] > # DremioSqlToRelConverter > > [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/planner/sql/DremioSqlToRelConverter.java#L40] > # (Dremio) SqlValidatorImpl > > [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/planner/sql/SqlValidatorImpl.java#L32] > # Dre
[jira] [Updated] (CALCITE-2866) Allow passing factory of SqlValidator / SqlToRelConverter / CatalogReader into FrameworkConfig
[ https://issues.apache.org/jira/browse/CALCITE-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongze Zhang updated CALCITE-2866: -- Summary: Allow passing factory of SqlValidator / SqlToRelConverter / CatalogReader into FrameworkConfig (was: Allow passing factory of SqlValidator / SqlToRelConverter into FrameworkConfig) > Allow passing factory of SqlValidator / SqlToRelConverter / CatalogReader > into FrameworkConfig > --- > > Key: CALCITE-2866 > URL: https://issues.apache.org/jira/browse/CALCITE-2866 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Hongze Zhang >Assignee: Hongze Zhang >Priority: Major > Labels: usability > > It seems that the customization of SqlValidator / SqlToRelConverter / > CatalogReader in downstream projects is normal. > E.g. Following are some examples (not all) of extended SqlValidator / > SqlToRelConverter / CatalogReader from other projects that are using Calcite > as query planner : > # DrillValidator > > [https://github.com/apache/drill/blob/4627973bde9847a4eb2672c44941136c167326a1/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java#L249] > # FlinkCalciteSqlValidator > > [https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/calcite/FlinkCalciteSqlValidator.scala#L31] > # DremioSqlToRelConverter > > [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/planner/sql/DremioSqlToRelConverter.java#L40] > # (Dremio) SqlValidatorImpl > > [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/planner/sql/SqlValidatorImpl.java#L32] > # DremioCatalogReader > > [https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/catalog/DremioCatalogReader.java#L71] > The class tools/Planner.java[1] is a good customization tool for Calcite > user. However currently there are no way to use Frameworks.getPlanner() > directly if the classes need to be customized[2][3]. > > [1] > [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/Planner.java] > [2] > [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L232] > [3] > [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L184] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2269) Enable Error Prone checking
[ https://issues.apache.org/jira/browse/CALCITE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803928#comment-16803928 ] Kevin Risden commented on CALCITE-2269: --- Merged in commit 1b68d1fae721f45405e580582abb8fede9a6c028 > Enable Error Prone checking > --- > > Key: CALCITE-2269 > URL: https://issues.apache.org/jira/browse/CALCITE-2269 > Project: Calcite > Issue Type: Improvement > Components: build >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Major > Labels: pull-request-available > Fix For: avatica-1.14.0 > > Time Spent: 50m > Remaining Estimate: 0h > > [http://errorprone.info/] provides the ability to do static checks like > Spotbugs (CALCITE-1006). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CALCITE-2269) Enable Error Prone checking
[ https://issues.apache.org/jira/browse/CALCITE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Risden resolved CALCITE-2269. --- Resolution: Fixed > Enable Error Prone checking > --- > > Key: CALCITE-2269 > URL: https://issues.apache.org/jira/browse/CALCITE-2269 > Project: Calcite > Issue Type: Improvement > Components: build >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Major > Labels: pull-request-available > Fix For: avatica-1.14.0 > > Time Spent: 50m > Remaining Estimate: 0h > > [http://errorprone.info/] provides the ability to do static checks like > Spotbugs (CALCITE-1006). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2865) FilterProjectTransposeRule generates wrong traitSet when copyFilter/Project is true
[ https://issues.apache.org/jira/browse/CALCITE-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804041#comment-16804041 ] Haisheng Yuan commented on CALCITE-2865: I am curious why does it provide copyFilter and copyProject option. > FilterProjectTransposeRule generates wrong traitSet when copyFilter/Project > is true > --- > > Key: CALCITE-2865 > URL: https://issues.apache.org/jira/browse/CALCITE-2865 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.18.0 >Reporter: Ruben Quesada Lopez >Assignee: Ruben Quesada Lopez >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Problem can be reproduced with the following steps: > As suggested by [~zabetak], a possibility to reproduce this issue is with the > following test in RelOptRulesTest: > {code:java} > /** Test case for >* href="https://issues.apache.org/jira/browse/CALCITE-2865";>[CALCITE-2865] >* FilterProjectTransposeRule generates wrong traitSet when > copyFilter/Project is true. */ > @Test public void testFilterProjectTransposeRule() { > List rules = Arrays.asList( > FilterProjectTransposeRule.INSTANCE, // default values: > copyFilter=true, copyProject=true > new FilterProjectTransposeRule(Filter.class, Project.class, > /*copyFilter*/ false, /*copyProject*/ false, RelFactories.LOGICAL_BUILDER)); > List results = new ArrayList<>(2); > for (RelOptRule rule : rules) { > RelBuilder b = RelBuilder.create(RelBuilderTest.config().build()); > RelNode in = b > .scan("EMP") > .sort(-4) // salary desc > .project(b.field(3)) // salary > .filter(b.equals(b.field(0), b.literal(11500))) // salary = > 11500 > .build(); > HepProgram program = new HepProgramBuilder() > .addRuleInstance(rule) > .build(); > HepPlanner hepPlanner = new HepPlanner(program); > hepPlanner.setRoot(in); > results.add(hepPlanner.findBestExp()); > } > // compare LogicalFilter traitSet > assertEquals(results.get(0).getInput(0).getTraitSet(), > results.get(1).getInput(0).getTraitSet()); > } > {code} > Which fails with the following message: > {code:java} > java.lang.AssertionError: > Expected :NONE.[0 DESC] > Actual :NONE.[3 DESC] > {code} > If we run the test with a break point at the end of > {{FilterProjectTransposeRule#onMatch}} method, we can see that, when the > project and filter are transposed, the generated output (newFilterRel) is > different in terms of traitSet, depending if the copyFilter / copyProject > flags are used or not: > - copyFilter and copyProject TRUE (default) > {code:java} > LogicalProject.NONE.[0 DESC] > LogicalFilter.NONE.[0 DESC] // !!! > LogicalSort.NONE.[3 DESC] > {code} > - copyFilter and copyProject FALSE > {code:java} > LogicalProject.NONE.[0 DESC] > LogicalFilter.NONE.[3 DESC] // !!! > LogicalSort.NONE.[3 DESC] > {code} > As I see it, the default behavior (copyFilter/Project true) seems wrong in > terms of new filter's traitSet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-1581) UDTF like in hive
[ https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-1581: Labels: pull-request-available udtf (was: udtf) > UDTF like in hive > - > > Key: CALCITE-1581 > URL: https://issues.apache.org/jira/browse/CALCITE-1581 > Project: Calcite > Issue Type: New Feature >Reporter: Xiaoyong Deng >Assignee: Sihua Zhou >Priority: Major > Labels: pull-request-available, udtf > > Support one row in and multi-column/multi-row out(one-to-many mapping), just > like udtf in hive. > The query would like this: > {code} > select > func(c0, c1) as (f0, f1, f2) > from table_name; > {code} > c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-1581) UDTF like in hive
[ https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804052#comment-16804052 ] pengzhiwei commented on CALCITE-1581: - Hi [~julianhyde], I have make a pr [https://github.com/apache/calcite/pull/1138] for this issue. In this pr,I mainly do the follow things: * Rewrite the "parser.jj" file to support "udtf() as (f0,f1)" in the sql parser stage with minimum modification. * add allowSelectTableFunction() in SqlConformance and also add a kind of Hive SqlConformance. * In SqlValidator stage, I rewrite the "udtf() as (f0,f1)" to the lateral-table sql-node. With this rewritting,this is nothing need to do in the SqlToRelConverter. > UDTF like in hive > - > > Key: CALCITE-1581 > URL: https://issues.apache.org/jira/browse/CALCITE-1581 > Project: Calcite > Issue Type: New Feature >Reporter: Xiaoyong Deng >Assignee: Sihua Zhou >Priority: Major > Labels: pull-request-available, udtf > Time Spent: 10m > Remaining Estimate: 0h > > Support one row in and multi-column/multi-row out(one-to-many mapping), just > like udtf in hive. > The query would like this: > {code} > select > func(c0, c1) as (f0, f1, f2) > from table_name; > {code} > c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-1515) Support TableFunctionScan in RelBuilder
[ https://issues.apache.org/jira/browse/CALCITE-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804165#comment-16804165 ] Julian Hyde commented on CALCITE-1515: -- [~Chunwei Lei], I did some fix-up (removing the redundant 'inputCount' argument to CURSOR, and allowing a table function to be called with zero inputs). Please take a look at https://github.com/julianhyde/calcite/commit/bd116f45fb9438293a55260717500391bf977197; if you +1 I will squash and commit. > Support TableFunctionScan in RelBuilder > --- > > Key: CALCITE-1515 > URL: https://issues.apache.org/jira/browse/CALCITE-1515 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Anton Mushin >Assignee: Chunwei Lei >Priority: Minor > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > Calcite has a TableFunctionScan, which is a RelNode, takes a function, 0 or > more RelNode inputs, and 0 or more other arguments. > RelBuilder does not support TableFunctionScan yet. > add In RelBuilder TableFunctionScanFactory which will create > LogicalTableFunctionScan like as for LogicalTableScan and factory for others > RelNodes [ > [example|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L125] > ] > also adding a functionScan method, analogous to the scan method but for table > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2921) nullif(null,y) throws exception in verification
[ https://issues.apache.org/jira/browse/CALCITE-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804256#comment-16804256 ] Julian Hyde commented on CALCITE-2921: -- I have reviewed PR 1117: * Could you use java.lang.Void rather than adding a new class Null? * Can you enable this behavior with a new method {{SqlConformance.allowNakedNull()}} * Also add a test for {{nullif(null,y) }} since this is the title of this bug * Change {{INSTANCE_COUNT}} to a {{ThreadLocal}} since the function might be used by several tests in parallel > nullif(null,y) throws exception in verification > > > Key: CALCITE-2921 > URL: https://issues.apache.org/jira/browse/CALCITE-2921 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently calcite will translate "nullif(x,y)" to a case-when expression,just > like "case when x = y then null else x".So when "x" is null literal,a > exception throws out as follow: > {code:java} > ELSE clause or at least one THEN clause must be non-NULL > {code} > I have test in mysql,"nullif(null,y) works well.So I think we should allow > this usage of "nullif". > There are two ways to fix this issue: > 1) Skip the check for "foundNotNull" in SqlCaseOperator#checkOperandTypes: > > {code:java} > if (!foundNotNull) { > // according to the sql standard we can not have all of the THEN > // statements and the ELSE returning null > if (throwOnFailure) { > throw callBinding.newError(RESOURCE.mustNotNullInElse()); > } > return false; > }{code} > However, as the comment says, we cannot have all of the THEN and ELSE > returning null. > 2) Disable the translation from nullif to case-when and keep "nullif" as it > is. > Any suggestion is welcomed,Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CALCITE-2658) Introducing more ReduceExpressionRules
[ https://issues.apache.org/jira/browse/CALCITE-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-2658. Resolution: Fixed Fix Version/s: 1.20.0 > Introducing more ReduceExpressionRules > -- > > Key: CALCITE-2658 > URL: https://issues.apache.org/jira/browse/CALCITE-2658 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > It is useful to have rules reducing Exchange/Sort/SortExchange keys, e.g., > SELECT key,value FROM (SELECT 1 AS key, value FROM src) r DISTRIBUTE BY key; > can be reduced to > SELECT 1 AS key, value FROM src;# Since singleton requirement may already > required by SELECT. > SELECT key,value FROM (SELECT 1 AS key, value FROM src) r ORDER BY key; > can be reduced to > SELECT 1 AS key, value FROM src;# Since ordering on constant is useless. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-1515) Support TableFunctionScan in RelBuilder
[ https://issues.apache.org/jira/browse/CALCITE-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804513#comment-16804513 ] Chunwei Lei commented on CALCITE-1515: -- [~julianhyde] +1, thank you for your review and fix-up. > Support TableFunctionScan in RelBuilder > --- > > Key: CALCITE-1515 > URL: https://issues.apache.org/jira/browse/CALCITE-1515 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Anton Mushin >Assignee: Chunwei Lei >Priority: Minor > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > Calcite has a TableFunctionScan, which is a RelNode, takes a function, 0 or > more RelNode inputs, and 0 or more other arguments. > RelBuilder does not support TableFunctionScan yet. > add In RelBuilder TableFunctionScanFactory which will create > LogicalTableFunctionScan like as for LogicalTableScan and factory for others > RelNodes [ > [example|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L125] > ] > also adding a functionScan method, analogous to the scan method but for table > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2960) CalciteCatalogReader use a hard-coding config to get functions
[ https://issues.apache.org/jira/browse/CALCITE-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804519#comment-16804519 ] Lai Zhou edited comment on CALCITE-2960 at 3/29/19 1:57 AM: [~danny0405], it is really a part of issue 2928. In my case ,I use the FrameworkConfig to create a sql planner,the config is like: {code:java} Frameworks .newConfigBuilder() .defaultSchema(rootSchema) .parserConfig(sqlParserConfig) .operatorTable(sqlStdOperatorTable) .context(Contexts.of(calciteConnectionConfig)) .build(); {code} User-defined functions will be added into the rootSchema , and a CalciteCatalogReader will be created based on this rootSchema. {code:java} new CalciteCatalogReader( CalciteSchema.from(rootSchema), CalciteSchema.from(rootSchema).path(null), new JavaTypeFactoryImpl(typeSystem), calciteConnectionConfig); {code} Then the FrameworkConfig will add the CalciteCatalogReader to it's ChainedSqlOperatorTable. {code:java} createFrameworkConfig(sqlParserConfig, ChainedSqlOperatorTable.of(sqlStdOperatorTable, calciteCatalogReader), calciteConnectionConfig); {code} So in the runtime , the sql planner will look up operators from the CalciteCatalogReader . was (Author: hhlai1990): [~danny0405], it is really a part of issue 2928. In my case ,I use the FrameworkConfig to create a sql planner,the config is like: {code:java} Frameworks .newConfigBuilder() .defaultSchema(rootSchema) .parserConfig(sqlParserConfig) .operatorTable(sqlStdOperatorTable) .context(Contexts.of(calciteConnectionConfig)) .build(); {code} User-defined functions will be added into the rootSchema , and a CalciteCatalogReader will be created based on this rootSchema. {code:java} new CalciteCatalogReader( CalciteSchema.from(rootSchema), CalciteSchema.from(rootSchema).path(null), new JavaTypeFactoryImpl(typeSystem), calciteConnectionConfig); {code} Then the FrameworkConfig will add the CalciteCatalogReader to it's ChainedSqlOperatorTable. {code:java} createFrameworkConfig(sqlParserConfig, ChainedSqlOperatorTable.of(sqlStdOperatorTable, calciteCatalogReader), calciteConnectionConfig); {code} So in the runtime , the sql planner will look up operators from the CalciteCatalogReader . > CalciteCatalogReader use a hard-coding config to get functions > -- > > Key: CALCITE-2960 > URL: https://issues.apache.org/jira/browse/CALCITE-2960 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Lai Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CalciteCatalogReader use a hard-coding config to get functions > {code:java} > functions2.addAll(schema.getFunctions(name, true)); > {code} > the right way is: > {code:java} > functions2.addAll(schema.getFunctions(name, config.caseSensitive())); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2960) CalciteCatalogReader use a hard-coding config to get functions
[ https://issues.apache.org/jira/browse/CALCITE-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804519#comment-16804519 ] Lai Zhou commented on CALCITE-2960: --- [~danny0405], it is really a part of issue 2928. In my case ,I use the FrameworkConfig to create a sql planner,the config is like: {code:java} Frameworks .newConfigBuilder() .defaultSchema(rootSchema) .parserConfig(sqlParserConfig) .operatorTable(sqlStdOperatorTable) .context(Contexts.of(calciteConnectionConfig)) .build(); {code} User-defined functions will be added into the rootSchema , and a CalciteCatalogReader will be created based on this rootSchema. {code:java} new CalciteCatalogReader( CalciteSchema.from(rootSchema), CalciteSchema.from(rootSchema).path(null), new JavaTypeFactoryImpl(typeSystem), calciteConnectionConfig); {code} Then the FrameworkConfig will add the CalciteCatalogReader to it's ChainedSqlOperatorTable. {code:java} createFrameworkConfig(sqlParserConfig, ChainedSqlOperatorTable.of(sqlStdOperatorTable, calciteCatalogReader), calciteConnectionConfig); {code} So in the runtime , the sql planner will look up operators from the CalciteCatalogReader . > CalciteCatalogReader use a hard-coding config to get functions > -- > > Key: CALCITE-2960 > URL: https://issues.apache.org/jira/browse/CALCITE-2960 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Lai Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CalciteCatalogReader use a hard-coding config to get functions > {code:java} > functions2.addAll(schema.getFunctions(name, true)); > {code} > the right way is: > {code:java} > functions2.addAll(schema.getFunctions(name, config.caseSensitive())); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2960) CalciteCatalogReader use a hard-coding config to get functions
[ https://issues.apache.org/jira/browse/CALCITE-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804519#comment-16804519 ] Lai Zhou edited comment on CALCITE-2960 at 3/29/19 1:59 AM: [~danny0405], it is really a part of issue 2928. In my case ,I use the FrameworkConfig to create a sql planner,the config is like: {code:java} Frameworks .newConfigBuilder() .defaultSchema(rootSchema) .parserConfig(sqlParserConfig) .operatorTable(sqlOperatorTable) .context(Contexts.of(calciteConnectionConfig)) .build(); {code} User-defined functions will be added into the rootSchema , and a CalciteCatalogReader will be created based on this rootSchema. {code:java} new CalciteCatalogReader( CalciteSchema.from(rootSchema), CalciteSchema.from(rootSchema).path(null), new JavaTypeFactoryImpl(typeSystem), calciteConnectionConfig); {code} Then the FrameworkConfig will add the CalciteCatalogReader to it's ChainedSqlOperatorTable. {code:java} createFrameworkConfig(sqlParserConfig, ChainedSqlOperatorTable.of(sqlStdOperatorTable, calciteCatalogReader), calciteConnectionConfig); {code} So in the runtime , the sql planner will look up operators from the CalciteCatalogReader . was (Author: hhlai1990): [~danny0405], it is really a part of issue 2928. In my case ,I use the FrameworkConfig to create a sql planner,the config is like: {code:java} Frameworks .newConfigBuilder() .defaultSchema(rootSchema) .parserConfig(sqlParserConfig) .operatorTable(sqlStdOperatorTable) .context(Contexts.of(calciteConnectionConfig)) .build(); {code} User-defined functions will be added into the rootSchema , and a CalciteCatalogReader will be created based on this rootSchema. {code:java} new CalciteCatalogReader( CalciteSchema.from(rootSchema), CalciteSchema.from(rootSchema).path(null), new JavaTypeFactoryImpl(typeSystem), calciteConnectionConfig); {code} Then the FrameworkConfig will add the CalciteCatalogReader to it's ChainedSqlOperatorTable. {code:java} createFrameworkConfig(sqlParserConfig, ChainedSqlOperatorTable.of(sqlStdOperatorTable, calciteCatalogReader), calciteConnectionConfig); {code} So in the runtime , the sql planner will look up operators from the CalciteCatalogReader . > CalciteCatalogReader use a hard-coding config to get functions > -- > > Key: CALCITE-2960 > URL: https://issues.apache.org/jira/browse/CALCITE-2960 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Lai Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CalciteCatalogReader use a hard-coding config to get functions > {code:java} > functions2.addAll(schema.getFunctions(name, true)); > {code} > the right way is: > {code:java} > functions2.addAll(schema.getFunctions(name, config.caseSensitive())); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CALCITE-2744) Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query with a multi-param aggregate function in subquery
[ https://issues.apache.org/jira/browse/CALCITE-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chan reassigned CALCITE-2744: --- Assignee: Danny Chan > Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query > with a multi-param aggregate function in subquery > -- > > Key: CALCITE-2744 > URL: https://issues.apache.org/jira/browse/CALCITE-2744 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.17.0 >Reporter: Wei Zhong >Assignee: Danny Chan >Priority: Major > > when running the following test in org.apache.calcite.test.RelOptRulesTest, > we will get a RuntimeException: > {code:java} > @Test public void testDecorrelateWith2ParamsAgg() { > final HepProgram preProgram = > HepProgram.builder() > .addRuleInstance(FilterProjectTransposeRule.INSTANCE) > .addRuleInstance(FilterJoinRule.FILTER_ON_JOIN) > .addRuleInstance(ProjectMergeRule.INSTANCE) > .build(); > final HepProgram program = > HepProgram.builder() > .addRuleInstance(SemiJoinRule.PROJECT) > .build(); > final String sql = "select * from dept where exists (\n" > + "select UDF_AVG(sal, sal + 1) from sales.emp\n" > + "where emp.deptno = dept.deptno\n" > + "and emp.sal > 100)"; > Sql sqlTest = sql(sql) > .withDecorrelation(true) > .withTrim(true) > .withPre(preProgram) > .with(program); > sqlTest.check(); > } > {code} > UDF_AVG code in MockSqlOperatorTable: > {code:java} > public static void addRamp(MockSqlOperatorTable opTab) { > // Don't use anonymous inner classes. They can't be instantiated > // using reflection when we are deserializing from JSON. > opTab.addOperator(new RampFunction()); > opTab.addOperator(new DedupFunction()); > opTab.addOperator(UDF_AVG); > } > public static final SqlFunction UDF_AVG = new SqlAggFunction( > "UDF_AVG", > null, > SqlKind.OTHER_FUNCTION, > ReturnTypes.AVG_AGG_FUNCTION, > null, > OperandTypes.family(SqlTypeFamily.NUMERIC, SqlTypeFamily.NUMERIC), > SqlFunctionCategory.NUMERIC, > false, > false, > Optionality.FORBIDDEN) { > @Override public boolean isDeterministic() { > return false; > } > }; > {code} > The RuntimeExcpetion detail: > {code:java} > java.lang.RuntimeException: While invoking method 'public > org.apache.calcite.sql2rel.RelDecorrelator$Frame > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(org.apache.calcite.rel.logical.LogicalProject)' > at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527) > at > org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:613) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:254) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:214) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelateQuery(SqlToRelConverter.java:3037) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelate(SqlToRelConverter.java:481) > at > org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:615) > at > org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:177) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:334) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:317) > at > org.apache.calcite.test.RelOptRulesTest.testDecorrelateWith2ParamsAgg(RelOptRulesTest.java:4286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > a
[jira] [Updated] (CALCITE-2744) Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query with a multi-param aggregate function in subquery
[ https://issues.apache.org/jira/browse/CALCITE-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chan updated CALCITE-2744: Fix Version/s: 1.20.0 > Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query > with a multi-param aggregate function in subquery > -- > > Key: CALCITE-2744 > URL: https://issues.apache.org/jira/browse/CALCITE-2744 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.17.0 >Reporter: Wei Zhong >Assignee: Danny Chan >Priority: Major > Fix For: 1.20.0 > > > when running the following test in org.apache.calcite.test.RelOptRulesTest, > we will get a RuntimeException: > {code:java} > @Test public void testDecorrelateWith2ParamsAgg() { > final HepProgram preProgram = > HepProgram.builder() > .addRuleInstance(FilterProjectTransposeRule.INSTANCE) > .addRuleInstance(FilterJoinRule.FILTER_ON_JOIN) > .addRuleInstance(ProjectMergeRule.INSTANCE) > .build(); > final HepProgram program = > HepProgram.builder() > .addRuleInstance(SemiJoinRule.PROJECT) > .build(); > final String sql = "select * from dept where exists (\n" > + "select UDF_AVG(sal, sal + 1) from sales.emp\n" > + "where emp.deptno = dept.deptno\n" > + "and emp.sal > 100)"; > Sql sqlTest = sql(sql) > .withDecorrelation(true) > .withTrim(true) > .withPre(preProgram) > .with(program); > sqlTest.check(); > } > {code} > UDF_AVG code in MockSqlOperatorTable: > {code:java} > public static void addRamp(MockSqlOperatorTable opTab) { > // Don't use anonymous inner classes. They can't be instantiated > // using reflection when we are deserializing from JSON. > opTab.addOperator(new RampFunction()); > opTab.addOperator(new DedupFunction()); > opTab.addOperator(UDF_AVG); > } > public static final SqlFunction UDF_AVG = new SqlAggFunction( > "UDF_AVG", > null, > SqlKind.OTHER_FUNCTION, > ReturnTypes.AVG_AGG_FUNCTION, > null, > OperandTypes.family(SqlTypeFamily.NUMERIC, SqlTypeFamily.NUMERIC), > SqlFunctionCategory.NUMERIC, > false, > false, > Optionality.FORBIDDEN) { > @Override public boolean isDeterministic() { > return false; > } > }; > {code} > The RuntimeExcpetion detail: > {code:java} > java.lang.RuntimeException: While invoking method 'public > org.apache.calcite.sql2rel.RelDecorrelator$Frame > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(org.apache.calcite.rel.logical.LogicalProject)' > at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527) > at > org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:613) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:254) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:214) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelateQuery(SqlToRelConverter.java:3037) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelate(SqlToRelConverter.java:481) > at > org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:615) > at > org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:177) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:334) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:317) > at > org.apache.calcite.test.RelOptRulesTest.testDecorrelateWith2ParamsAgg(RelOptRulesTest.java:4286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000
[jira] [Commented] (CALCITE-2744) Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query with a multi-param aggregate function in subquery
[ https://issues.apache.org/jira/browse/CALCITE-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804543#comment-16804543 ] Danny Chan commented on CALCITE-2744: - It seems that it is an edge case that an agg function can take 2 parameters, fix of just 2 is not that necessary, after all there are maybe 3 or more parameters, so i kind of not fix this and just let it throw exception. > Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query > with a multi-param aggregate function in subquery > -- > > Key: CALCITE-2744 > URL: https://issues.apache.org/jira/browse/CALCITE-2744 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.17.0 >Reporter: Wei Zhong >Assignee: Danny Chan >Priority: Major > > when running the following test in org.apache.calcite.test.RelOptRulesTest, > we will get a RuntimeException: > {code:java} > @Test public void testDecorrelateWith2ParamsAgg() { > final HepProgram preProgram = > HepProgram.builder() > .addRuleInstance(FilterProjectTransposeRule.INSTANCE) > .addRuleInstance(FilterJoinRule.FILTER_ON_JOIN) > .addRuleInstance(ProjectMergeRule.INSTANCE) > .build(); > final HepProgram program = > HepProgram.builder() > .addRuleInstance(SemiJoinRule.PROJECT) > .build(); > final String sql = "select * from dept where exists (\n" > + "select UDF_AVG(sal, sal + 1) from sales.emp\n" > + "where emp.deptno = dept.deptno\n" > + "and emp.sal > 100)"; > Sql sqlTest = sql(sql) > .withDecorrelation(true) > .withTrim(true) > .withPre(preProgram) > .with(program); > sqlTest.check(); > } > {code} > UDF_AVG code in MockSqlOperatorTable: > {code:java} > public static void addRamp(MockSqlOperatorTable opTab) { > // Don't use anonymous inner classes. They can't be instantiated > // using reflection when we are deserializing from JSON. > opTab.addOperator(new RampFunction()); > opTab.addOperator(new DedupFunction()); > opTab.addOperator(UDF_AVG); > } > public static final SqlFunction UDF_AVG = new SqlAggFunction( > "UDF_AVG", > null, > SqlKind.OTHER_FUNCTION, > ReturnTypes.AVG_AGG_FUNCTION, > null, > OperandTypes.family(SqlTypeFamily.NUMERIC, SqlTypeFamily.NUMERIC), > SqlFunctionCategory.NUMERIC, > false, > false, > Optionality.FORBIDDEN) { > @Override public boolean isDeterministic() { > return false; > } > }; > {code} > The RuntimeExcpetion detail: > {code:java} > java.lang.RuntimeException: While invoking method 'public > org.apache.calcite.sql2rel.RelDecorrelator$Frame > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(org.apache.calcite.rel.logical.LogicalProject)' > at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527) > at > org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:613) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:254) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:214) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelateQuery(SqlToRelConverter.java:3037) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelate(SqlToRelConverter.java:481) > at > org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:615) > at > org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:177) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:334) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:317) > at > org.apache.calcite.test.RelOptRulesTest.testDecorrelateWith2ParamsAgg(RelOptRulesTest.java:4286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.Paren
[jira] [Updated] (CALCITE-2744) Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query with a multi-param aggregate function in subquery
[ https://issues.apache.org/jira/browse/CALCITE-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chan updated CALCITE-2744: Fix Version/s: (was: 1.20.0) > Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query > with a multi-param aggregate function in subquery > -- > > Key: CALCITE-2744 > URL: https://issues.apache.org/jira/browse/CALCITE-2744 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.17.0 >Reporter: Wei Zhong >Assignee: Danny Chan >Priority: Major > > when running the following test in org.apache.calcite.test.RelOptRulesTest, > we will get a RuntimeException: > {code:java} > @Test public void testDecorrelateWith2ParamsAgg() { > final HepProgram preProgram = > HepProgram.builder() > .addRuleInstance(FilterProjectTransposeRule.INSTANCE) > .addRuleInstance(FilterJoinRule.FILTER_ON_JOIN) > .addRuleInstance(ProjectMergeRule.INSTANCE) > .build(); > final HepProgram program = > HepProgram.builder() > .addRuleInstance(SemiJoinRule.PROJECT) > .build(); > final String sql = "select * from dept where exists (\n" > + "select UDF_AVG(sal, sal + 1) from sales.emp\n" > + "where emp.deptno = dept.deptno\n" > + "and emp.sal > 100)"; > Sql sqlTest = sql(sql) > .withDecorrelation(true) > .withTrim(true) > .withPre(preProgram) > .with(program); > sqlTest.check(); > } > {code} > UDF_AVG code in MockSqlOperatorTable: > {code:java} > public static void addRamp(MockSqlOperatorTable opTab) { > // Don't use anonymous inner classes. They can't be instantiated > // using reflection when we are deserializing from JSON. > opTab.addOperator(new RampFunction()); > opTab.addOperator(new DedupFunction()); > opTab.addOperator(UDF_AVG); > } > public static final SqlFunction UDF_AVG = new SqlAggFunction( > "UDF_AVG", > null, > SqlKind.OTHER_FUNCTION, > ReturnTypes.AVG_AGG_FUNCTION, > null, > OperandTypes.family(SqlTypeFamily.NUMERIC, SqlTypeFamily.NUMERIC), > SqlFunctionCategory.NUMERIC, > false, > false, > Optionality.FORBIDDEN) { > @Override public boolean isDeterministic() { > return false; > } > }; > {code} > The RuntimeExcpetion detail: > {code:java} > java.lang.RuntimeException: While invoking method 'public > org.apache.calcite.sql2rel.RelDecorrelator$Frame > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(org.apache.calcite.rel.logical.LogicalProject)' > at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527) > at > org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:613) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:254) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:214) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelateQuery(SqlToRelConverter.java:3037) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelate(SqlToRelConverter.java:481) > at > org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:615) > at > org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:177) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:334) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:317) > at > org.apache.calcite.test.RelOptRulesTest.testDecorrelateWith2ParamsAgg(RelOptRulesTest.java:4286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58
[jira] [Closed] (CALCITE-2744) Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query with a multi-param aggregate function in subquery
[ https://issues.apache.org/jira/browse/CALCITE-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chan closed CALCITE-2744. --- Resolution: Won't Fix > Throws RuntimeException in RelDecorrelator when optimizing a Semi-Join query > with a multi-param aggregate function in subquery > -- > > Key: CALCITE-2744 > URL: https://issues.apache.org/jira/browse/CALCITE-2744 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.17.0 >Reporter: Wei Zhong >Assignee: Danny Chan >Priority: Major > > when running the following test in org.apache.calcite.test.RelOptRulesTest, > we will get a RuntimeException: > {code:java} > @Test public void testDecorrelateWith2ParamsAgg() { > final HepProgram preProgram = > HepProgram.builder() > .addRuleInstance(FilterProjectTransposeRule.INSTANCE) > .addRuleInstance(FilterJoinRule.FILTER_ON_JOIN) > .addRuleInstance(ProjectMergeRule.INSTANCE) > .build(); > final HepProgram program = > HepProgram.builder() > .addRuleInstance(SemiJoinRule.PROJECT) > .build(); > final String sql = "select * from dept where exists (\n" > + "select UDF_AVG(sal, sal + 1) from sales.emp\n" > + "where emp.deptno = dept.deptno\n" > + "and emp.sal > 100)"; > Sql sqlTest = sql(sql) > .withDecorrelation(true) > .withTrim(true) > .withPre(preProgram) > .with(program); > sqlTest.check(); > } > {code} > UDF_AVG code in MockSqlOperatorTable: > {code:java} > public static void addRamp(MockSqlOperatorTable opTab) { > // Don't use anonymous inner classes. They can't be instantiated > // using reflection when we are deserializing from JSON. > opTab.addOperator(new RampFunction()); > opTab.addOperator(new DedupFunction()); > opTab.addOperator(UDF_AVG); > } > public static final SqlFunction UDF_AVG = new SqlAggFunction( > "UDF_AVG", > null, > SqlKind.OTHER_FUNCTION, > ReturnTypes.AVG_AGG_FUNCTION, > null, > OperandTypes.family(SqlTypeFamily.NUMERIC, SqlTypeFamily.NUMERIC), > SqlFunctionCategory.NUMERIC, > false, > false, > Optionality.FORBIDDEN) { > @Override public boolean isDeterministic() { > return false; > } > }; > {code} > The RuntimeExcpetion detail: > {code:java} > java.lang.RuntimeException: While invoking method 'public > org.apache.calcite.sql2rel.RelDecorrelator$Frame > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(org.apache.calcite.rel.logical.LogicalProject)' > at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527) > at > org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:613) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:254) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:214) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelateQuery(SqlToRelConverter.java:3037) > at > org.apache.calcite.sql2rel.SqlToRelConverter.decorrelate(SqlToRelConverter.java:481) > at > org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:615) > at > org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:177) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:334) > at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:317) > at > org.apache.calcite.test.RelOptRulesTest.testDecorrelateWith2ParamsAgg(RelOptRulesTest.java:4286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.ju
[jira] [Commented] (CALCITE-2921) nullif(null,y) throws exception in verification
[ https://issues.apache.org/jira/browse/CALCITE-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804546#comment-16804546 ] pengzhiwei commented on CALCITE-2921: - Thanks [~julianhyde], I will continue work. > nullif(null,y) throws exception in verification > > > Key: CALCITE-2921 > URL: https://issues.apache.org/jira/browse/CALCITE-2921 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently calcite will translate "nullif(x,y)" to a case-when expression,just > like "case when x = y then null else x".So when "x" is null literal,a > exception throws out as follow: > {code:java} > ELSE clause or at least one THEN clause must be non-NULL > {code} > I have test in mysql,"nullif(null,y) works well.So I think we should allow > this usage of "nullif". > There are two ways to fix this issue: > 1) Skip the check for "foundNotNull" in SqlCaseOperator#checkOperandTypes: > > {code:java} > if (!foundNotNull) { > // according to the sql standard we can not have all of the THEN > // statements and the ELSE returning null > if (throwOnFailure) { > throw callBinding.newError(RESOURCE.mustNotNullInElse()); > } > return false; > }{code} > However, as the comment says, we cannot have all of the THEN and ELSE > returning null. > 2) Disable the translation from nullif to case-when and keep "nullif" as it > is. > Any suggestion is welcomed,Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2921) nullif(null,y) throws exception in verification
[ https://issues.apache.org/jira/browse/CALCITE-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804546#comment-16804546 ] pengzhiwei edited comment on CALCITE-2921 at 3/29/19 3:07 AM: -- Thanks [~julianhyde], I will continue the work. was (Author: pzw2018): Thanks [~julianhyde], I will continue work. > nullif(null,y) throws exception in verification > > > Key: CALCITE-2921 > URL: https://issues.apache.org/jira/browse/CALCITE-2921 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently calcite will translate "nullif(x,y)" to a case-when expression,just > like "case when x = y then null else x".So when "x" is null literal,a > exception throws out as follow: > {code:java} > ELSE clause or at least one THEN clause must be non-NULL > {code} > I have test in mysql,"nullif(null,y) works well.So I think we should allow > this usage of "nullif". > There are two ways to fix this issue: > 1) Skip the check for "foundNotNull" in SqlCaseOperator#checkOperandTypes: > > {code:java} > if (!foundNotNull) { > // according to the sql standard we can not have all of the THEN > // statements and the ELSE returning null > if (throwOnFailure) { > throw callBinding.newError(RESOURCE.mustNotNullInElse()); > } > return false; > }{code} > However, as the comment says, we cannot have all of the THEN and ELSE > returning null. > 2) Disable the translation from nullif to case-when and keep "nullif" as it > is. > Any suggestion is welcomed,Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2960) CalciteCatalogReader use a hard-coding config to get functions
[ https://issues.apache.org/jira/browse/CALCITE-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804559#comment-16804559 ] Danny Chan commented on CALCITE-2960: - [~hhlai1990] I didn't found any method in calcite named createFrameworkConfig Can you past a code link here ? > CalciteCatalogReader use a hard-coding config to get functions > -- > > Key: CALCITE-2960 > URL: https://issues.apache.org/jira/browse/CALCITE-2960 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Lai Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CalciteCatalogReader use a hard-coding config to get functions > {code:java} > functions2.addAll(schema.getFunctions(name, true)); > {code} > the right way is: > {code:java} > functions2.addAll(schema.getFunctions(name, config.caseSensitive())); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2960) CalciteCatalogReader use a hard-coding config to get functions
[ https://issues.apache.org/jira/browse/CALCITE-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804563#comment-16804563 ] Lai Zhou commented on CALCITE-2960: --- It's my code. {code:java} protected FrameworkConfig createFrameworkConfig( SqlParser.Config sqlParserConfig, SqlOperatorTable sqlOperatorTable, CalciteConnectionConfig calciteConnectionConfig) { return Frameworks .newConfigBuilder() .defaultSchema(rootSchema) .parserConfig(sqlParserConfig) .operatorTable(sqlOperatorTable) .context(Contexts.of(calciteConnectionConfig)) .build(); } {code} I just use a customized ChainedSqlOperatorTable that includes a CalciteCatalogReader, the CalciteCatalogReader holds the rootSchema I created. So I can register a function into the rootSchema, when executing a sql query, the planner will look up an operator from the CalciteCatalogReader . Part of my code for parsing and converting the sql query is like: {code:java} try (Planner planner = Frameworks.getPlanner(frameworkConfig)) { RelRoot root; final SqlNode parsedSqlNode = planner.parse(sql); final Pair validatedSqlNodeAndType = planner .validateAndGetType( parsedSqlNode); root = planner.rel(validatedSqlNodeAndType.getKey()); ...{code} > CalciteCatalogReader use a hard-coding config to get functions > -- > > Key: CALCITE-2960 > URL: https://issues.apache.org/jira/browse/CALCITE-2960 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Lai Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CalciteCatalogReader use a hard-coding config to get functions > {code:java} > functions2.addAll(schema.getFunctions(name, true)); > {code} > the right way is: > {code:java} > functions2.addAll(schema.getFunctions(name, config.caseSensitive())); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2960) CalciteCatalogReader use a hard-coding config to get functions
[ https://issues.apache.org/jira/browse/CALCITE-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804563#comment-16804563 ] Lai Zhou edited comment on CALCITE-2960 at 3/29/19 3:53 AM: It's my code. {code:java} protected FrameworkConfig createFrameworkConfig( SqlParser.Config sqlParserConfig, SqlOperatorTable sqlOperatorTable, CalciteConnectionConfig calciteConnectionConfig) { return Frameworks .newConfigBuilder() .defaultSchema(rootSchema) .parserConfig(sqlParserConfig) .operatorTable(sqlOperatorTable) .context(Contexts.of(calciteConnectionConfig)) .build(); } {code} I just use a customized ChainedSqlOperatorTable that includes a CalciteCatalogReader, and the CalciteCatalogReader holds the rootSchema I created. So I can register a function into the rootSchema, when executing a sql query, the planner will look up an operator from the CalciteCatalogReader . Part of my code for parsing and converting the sql query is like: {code:java} try (Planner planner = Frameworks.getPlanner(frameworkConfig)) { RelRoot root; final SqlNode parsedSqlNode = planner.parse(sql); final Pair validatedSqlNodeAndType = planner .validateAndGetType( parsedSqlNode); root = planner.rel(validatedSqlNodeAndType.getKey()); ...{code} was (Author: hhlai1990): It's my code. {code:java} protected FrameworkConfig createFrameworkConfig( SqlParser.Config sqlParserConfig, SqlOperatorTable sqlOperatorTable, CalciteConnectionConfig calciteConnectionConfig) { return Frameworks .newConfigBuilder() .defaultSchema(rootSchema) .parserConfig(sqlParserConfig) .operatorTable(sqlOperatorTable) .context(Contexts.of(calciteConnectionConfig)) .build(); } {code} I just use a customized ChainedSqlOperatorTable that includes a CalciteCatalogReader, the CalciteCatalogReader holds the rootSchema I created. So I can register a function into the rootSchema, when executing a sql query, the planner will look up an operator from the CalciteCatalogReader . Part of my code for parsing and converting the sql query is like: {code:java} try (Planner planner = Frameworks.getPlanner(frameworkConfig)) { RelRoot root; final SqlNode parsedSqlNode = planner.parse(sql); final Pair validatedSqlNodeAndType = planner .validateAndGetType( parsedSqlNode); root = planner.rel(validatedSqlNodeAndType.getKey()); ...{code} > CalciteCatalogReader use a hard-coding config to get functions > -- > > Key: CALCITE-2960 > URL: https://issues.apache.org/jira/browse/CALCITE-2960 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Lai Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CalciteCatalogReader use a hard-coding config to get functions > {code:java} > functions2.addAll(schema.getFunctions(name, true)); > {code} > the right way is: > {code:java} > functions2.addAll(schema.getFunctions(name, config.caseSensitive())); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2960) CalciteCatalogReader use a hard-coding config to get functions
[ https://issues.apache.org/jira/browse/CALCITE-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804578#comment-16804578 ] Danny Chan commented on CALCITE-2960: - Got your idea, it's weird that Calcite itself never use CalciteCatalogReader as a SqlOperatorTable lookup, the only usage is for [1] which is used in a test case. Usually we will use a StdSqlOperatorTable chained with a ListSqlOperatorTable. But your usage is ok, after all, CalciteCatalogReader is definitely a SqlOperatorTable. Within CalciteCatalogReader, functions are managed by CalciteSchema, which are in the scope of a Catalog. [1] https://github.com/apache/calcite/blob/0537f27d87598dea888050a4cc63cf4d6fe027d7/core/src/main/java/org/apache/calcite/prepare/CalciteCatalogReader.java#L276 > CalciteCatalogReader use a hard-coding config to get functions > -- > > Key: CALCITE-2960 > URL: https://issues.apache.org/jira/browse/CALCITE-2960 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.18.0 >Reporter: Lai Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CalciteCatalogReader use a hard-coding config to get functions > {code:java} > functions2.addAll(schema.getFunctions(name, true)); > {code} > the right way is: > {code:java} > functions2.addAll(schema.getFunctions(name, config.caseSensitive())); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-1581) UDTF like in hive
[ https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804052#comment-16804052 ] pengzhiwei edited comment on CALCITE-1581 at 3/29/19 5:54 AM: -- Hi [~julianhyde], I have make a pr [https://github.com/apache/calcite/pull/1138] for this issue. In this pr,I mainly do the follow things: * Rewrite the "parser.jj" file to support "udtf() as (f0,f1)" in the sql parser stage with minimum modification. * add allowSelectTableFunction() in SqlConformance and also add a kind of Hive SqlConformance. * In SqlValidator stage, I rewrite the "udtf() as (f0,f1)" to the lateral-table sql-node. With this rewritting,there is nothing need to do in the SqlToRelConverter. was (Author: pzw2018): Hi [~julianhyde], I have make a pr [https://github.com/apache/calcite/pull/1138] for this issue. In this pr,I mainly do the follow things: * Rewrite the "parser.jj" file to support "udtf() as (f0,f1)" in the sql parser stage with minimum modification. * add allowSelectTableFunction() in SqlConformance and also add a kind of Hive SqlConformance. * In SqlValidator stage, I rewrite the "udtf() as (f0,f1)" to the lateral-table sql-node. With this rewritting,this is nothing need to do in the SqlToRelConverter. > UDTF like in hive > - > > Key: CALCITE-1581 > URL: https://issues.apache.org/jira/browse/CALCITE-1581 > Project: Calcite > Issue Type: New Feature >Reporter: Xiaoyong Deng >Assignee: Sihua Zhou >Priority: Major > Labels: pull-request-available, udtf > Time Spent: 2h > Remaining Estimate: 0h > > Support one row in and multi-column/multi-row out(one-to-many mapping), just > like udtf in hive. > The query would like this: > {code} > select > func(c0, c1) as (f0, f1, f2) > from table_name; > {code} > c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)