Alessandro Solimando created CALCITE-6429: ---------------------------------------------
Summary: Arrow adapter should default to the Enumerable convention for unsupported features Key: CALCITE-6429 URL: https://issues.apache.org/jira/browse/CALCITE-6429 Project: Calcite Issue Type: Sub-task Affects Versions: 1.37.0 Reporter: Alessandro Solimando Assignee: Alessandro Solimando Fix For: 1.38.0 Currently, the adapter fails with different errors when one of the known unsupported features is used, while it might simply catch the "UnsupportedOperationException" and bail out, thus allowing to generate a plan where the unsupported operation is implemented in the Enumerable convention. For instance, currently several tests are disabled for this reason, like [testArrowProjectFieldsWithDisjunctiveFilter|https://github.com/apache/calcite/blob/be9b860ebf3143dcbdbd0ee1777e604f0ace7b3c/arrow/src/test/java/org/apache/calcite/adapter/arrow/ArrowAdapterTest.java#L242], because it would fail as follows: {code:java} Error while executing SQL "select "intField", "stringField" from arrowdata where "intField"=12 or "stringField"='12'": Error while applying rule ArrowFilterRule, args [rel#31:LogicalFilter.NONE.[](input=RelSubset#15,condition=OR(=($0, 12), =($1, '12'))), rel#1:ArrowTableScan.ARROW.[](table=[ARROW, ARROWDATA],fields=[0, 1, 2, 3])] java.sql.SQLException: Error while executing SQL "select "intField", "stringField" from arrowdata where "intField"=12 or "stringField"='12'": Error while applying rule ArrowFilterRule, args [rel#31:LogicalFilter.NONE.[](input=RelSubset#15,condition=OR(=($0, 12), =($1, '12'))), rel#1:ArrowTableScan.ARROW.[](table=[ARROW, ARROWDATA],fields=[0, 1, 2, 3])] at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:164) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:228) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:566) at org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1495) at org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1434) at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1493) at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1483) at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1446) at org.apache.calcite.adapter.arrow.ArrowAdapterTest.testArrowProjectFieldsWithDisjunctiveFilter(ArrowAdapterTest.java:260) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185) at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Suppressed: org.apache.calcite.util.TestUtil$ExtraInformation: With materializationsEnabled=false, limit=-1 at org.apache.calcite.util.TestUtil.rethrow(TestUtil.java:389) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:598) ... 64 more Caused by: java.lang.RuntimeException: Error while applying rule ArrowFilterRule, args [rel#31:LogicalFilter.NONE.[](input=RelSubset#15,condition=OR(=($0, 12), =($1, '12'))), rel#1:ArrowTableScan.ARROW.[](table=[ARROW, ARROWDATA],fields=[0, 1, 2, 3])] at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250) at org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:59) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:524) at org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:282) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:348) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:177) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:220) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:666) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:519) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:487) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:237) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:702) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:677) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:157) ... 66 more Caused by: java.lang.UnsupportedOperationException: Unsupported disjunctive condition OR(=($0, 12), =($1, '12')) at org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70) at org.apache.calcite.adapter.arrow.ArrowFilter.<init>(ArrowFilter.java:43) at org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:103) at org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:88) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223) ... 80 more{code} The ideal plan the test should generate is: {code:java} "PLAN=ArrowToEnumerableConverter\n" + " ArrowProject(intField=[$0], stringField=[$1])\n" + " ArrowFilter(condition=[OR(=($0, 12), =($1, '12'))])\n" + " ArrowTableScan(table=[[ARROW, ARROWDATA]], fields=[[0, 1, 2, 3]])\n\n";{code} The test query, however, after the proposed workaround, could produce this valid plan in the meantime: {code:java} PLAN=EnumerableCalc(expr#0..1=[{inputs}], expr#2=[12], " + "expr#3=[=($t0, $t2)], expr#4=['12':VARCHAR], expr#5=[=($t1, $t4)], " + "expr#6=[OR($t3, $t5)], proj#0..1=[{exprs}], $condition=[$t6])\n" + " ArrowToEnumerableConverter\n" + " ArrowProject(intField=[$0], stringField=[$1])\n" + " ArrowTableScan(table=[[ARROW, ARROWDATA]], fields=[[0, 1, 2, 3]]){code} The advantage would be that more queries would be supported right away, while waiting for the missing features to be contributed. -- This message was sent by Atlassian Jira (v8.20.10#820010)