Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24747
to look at the new patch set (#2).
Change subject: IMPALA-15301: Export Impala's explicit cast as a Substrait cast
......................................................................
IMPALA-15301: Export Impala's explicit cast as a Substrait cast
ImpalaOperatorTable rewrites cast into an operator of its own, named
EXPLICIT_CAST and carrying SqlKind.OTHER, to keep Calcite from folding
it. The converter Isthmus already has for SqlKind.CAST therefore never
sees it, and every explicit cast stops the export. Over the TPC-DS
queries that carry Calcite goldens this is what blocks seven of them.
Add a call converter that emits a Substrait Cast, the same way
IMPALA-15298 handles IF.
The failure behaviour is the closest of the two Substrait offers rather
than an exact match, and that is worth stating plainly. Measured against
a running Impala: casting an unparseable string returns NULL, and an
integer that does not fit wraps round silently, so 99999999999 cast to
int comes back as 1215752191. Impala never raises. RETURN_NULL states
the first half. There is no Substrait failure behaviour for the second,
so a consumer would produce NULL where Impala produces the wrapped
value.
Almost every EXPLICIT_CAST in a plan is one the planner inserted to line
types up, where nothing overflows. A narrowing cast a user wrote is a
real difference, and the alternative is not exporting any plan that
contains a cast at all.
Two of the seven queries stop again further along, on a blocker the cast
had been hiding: one on the multiply that a datetime interval expands
into, one on SINGLE_VALUE. The corpus goes from 96 of 110 to 101 rather
than to 103.
SINGLE_VALUE gets a test of its own, saying why it stays refused rather
than reading as work not done yet. Calcite guards a scalar subquery it
cannot prove single-row with it, and Impala turns that into a
CardinalityCheckNode: the query fails at runtime if a second row arrives.
Substrait has any_value, which picks a row and asserts nothing, and no
single_value or assert_one_row. Mapping one onto the other would turn a
query that fails on bad data into one that quietly returns an arbitrary
row, with nothing in the plan for a consumer to notice.
Testing, in a local cluster on Linux:
- SubstraitPlanExporterTest: 18 tests, 0 failures, 0 errors. One added
test checks that the result is a CAST expression rather than a call,
that it carries RETURN_NULL, and that it targets the right type; the
other pins the SINGLE_VALUE refusal
- CalciteOptimizerTest and CalciteSubstraitCompatibilityTest, neither
of which this patch touches: 11 and 4 tests, 0 failures
- Withholding the converter fails the added test, so it is not vacuous
- A probe over the TPC-DS queries carrying Calcite goldens reports 101
of 110 with this patch and 96 without it
- git diff --check
Change-Id: I7523bc5ebd6239e92dc817bbe521920121ec8b2b
Assisted-by: Claude Opus 5 (Anthropic)
---
M
java/calcite-planner/src/test/java/org/apache/impala/calcite/service/SubstraitOperatorMappings.java
M
java/calcite-planner/src/test/java/org/apache/impala/calcite/service/SubstraitPlanExporterTest.java
2 files changed, 73 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/24747/2
--
To view, visit http://gerrit.cloudera.org:8080/24747
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7523bc5ebd6239e92dc817bbe521920121ec8b2b
Gerrit-Change-Number: 24747
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
