Aleksandr Efimov has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/24742 )
Change subject: IMPALA-15297: Map Impala operators onto Substrait functions ...................................................................... IMPALA-15297: Map Impala operators onto Substrait functions Twenty of the twenty-four shapes the direct harness covers export end to end. The rest stop on an operator Isthmus cannot resolve, and for most of them the reason is not that Substrait lacks the function. Isthmus resolves a scalar call by looking the operator up in a map keyed on the operator itself, and Calcite compares operators by class, name and kind. Impala substitutes its own classes for a good number of functions, so an operator that means exactly what Substrait's does still misses. Isthmus takes additional signatures for this: they say which Substrait function an Impala operator stands for and leave the operand matching alone. Add the signatures for the operators that have a counterpart: coalesce, concat, length, and floor and ceil with their d-prefixed spellings. Impala reaches these by three routes, the USE_IMPALA_OPERATOR set in ImpalaOperatorTable, builtins Calcite has no operator for, and the replacements in ImpalaCustomOperatorTable. All three resolve by name, so one table covers them. That takes the corpus from twenty to twenty-two. This is a list of what Isthmus has been taught, not the list of what it refuses that IMPALA-15296 deliberately does not keep. Rejection stays Isthmus's decision; every entry here is one a test covers. What is left needs a different mechanism rather than another name, and the table says so where a reader would otherwise read an omission. Substrait models a conditional as the IfThen expression rather than as a function, so IF has no name to map to. Substrait's round takes its digit count as i32 where Impala types the literal as i8, so the name lines up but the operands do not. Substrait has no truncate at all. LAG is a window function, resolved by a converter this table does not reach. Testing, in a local cluster on Linux, the three classes in one run: - SubstraitPlanExporterTest: 12 tests, 0 failures, 0 errors - CalciteOptimizerTest and CalciteSubstraitCompatibilityTest, neither of which this patch touches: 11 and 4 tests, 0 failures - Pointing the exporter back at the stock ConverterProvider fails the mapping test, so that assertion is not vacuous - A probe over the harness corpus reports 22 of 24 with this patch and 20 of 24 without it, the difference being the coalesce and the concat-with-length shapes - git diff --check Change-Id: I1824fa32d7cc294ee8e54c715bb4b40d37a152dd Assisted-by: Claude Opus 5 (Anthropic) --- A java/calcite-planner/src/test/java/org/apache/impala/calcite/service/SubstraitOperatorMappings.java M java/calcite-planner/src/test/java/org/apache/impala/calcite/service/SubstraitPlanExporter.java M java/calcite-planner/src/test/java/org/apache/impala/calcite/service/SubstraitPlanExporterTest.java 3 files changed, 164 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/24742/2 -- To view, visit http://gerrit.cloudera.org:8080/24742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1824fa32d7cc294ee8e54c715bb4b40d37a152dd Gerrit-Change-Number: 24742 Gerrit-PatchSet: 2 Gerrit-Owner: Aleksandr Efimov <[email protected]>
