[
https://issues.apache.org/jira/browse/IMPALA-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov updated IMPALA-15296:
--------------------------------------
Description:
The plan at CalciteOptimizer.createPreImpalaConvertPlan() is still Calcite's,
before it turns into Impala plan nodes. Export it as Substrait, with an outcome
that is either a plan or a reason there is none, never a half-converted one.
Relations are checked before conversion: Impala's own relations are
unambiguously outside Substrait, and listing them is more use than the
exception Isthmus raises on the first one it meets. Expressions are left to
Isthmus rather than to a list of what it refuses, which would drift and would
also be wrong, since Impala's aggregate classes convert while its scalar
functions do not.
What Isthmus cannot resolve on its own is taught to it. Isthmus looks a call up
in a map keyed on the operator and Calcite compares operators by class, name
and kind, so an Impala operator meaning exactly what Substrait's does still
misses. Signatures cover abs, coalesce, concat, length, floor and ceil with
their d-prefixed spellings, substr and round, the window functions lag,
first_value and last_value, and ndv. Three need more than a name: IF and cast
arrive as operators carrying SqlKind.OTHER and get converters of their own, and
substring, round and a datetime plus need their operands adjusted before a
variant matches.
What stays refused says why, in the table and in a test, so that a gap does not
read as an oversight: Substrait has no truncate and no one-argument round, its
approx_count_distinct takes one operand so a precision cannot be dropped
quietly, GROUPING has no Substrait function at all, and SINGLE_VALUE would have
to become any_value, which asserts nothing where Impala runs a cardinality
check.
This issue covers what was first filed as IMPALA-15297 through IMPALA-15302,
IMPALA-15305 and IMPALA-15307, which were folded into it before review.
was:
IMPALA-15282 added a test-only compatibility test that converts the
pre-conversion logical plan with Isthmus. It answers whether a handful of
covered shapes still convert. It does not give a caller a way to ask for the
conversion, or a usable answer when the conversion fails.
Add a test-scoped exporter over the same seam. It takes the plan and the
client-visible labels and returns either a Substrait plan or a rejection
carrying a reason, what Isthmus said, and the relations or operators the plan
contained. A half-converted plan is never returned.
What Isthmus accepts is decided by Isthmus rather than by a list of operators
kept in Impala. Such a list would drift as the operator table changes, and it
would also be wrong: Impala's aggregate classes convert because Isthmus
normalises aggregates by SqlKind, while its scalar functions resolve on
operator identity and do not, so the class name says nothing about whether an
operator exports.
Relations are the exception and are checked before conversion, because Impala's
own relations are unambiguously outside Substrait and listing them is more use
than the exception Isthmus raises on the first one it meets. That is the
cte_threshold case: above the threshold the repeated subtree becomes an
ImpalaSequence over a producer and one consumer per reference, and the seam
stops being a plan of Calcite relations.
The client-visible labels are passed in rather than read off the plan, because
the seam's row type is upper-cased and exporting its names would rename the
client's columns.
Isthmus stays test-scoped: no production dependency, no query option, nothing
added to impala-package.
> Calcite planner: export the pre-conversion logical plan to Substrait
> --------------------------------------------------------------------
>
> Key: IMPALA-15296
> URL: https://issues.apache.org/jira/browse/IMPALA-15296
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
> Labels: substrait
>
> The plan at CalciteOptimizer.createPreImpalaConvertPlan() is still Calcite's,
> before it turns into Impala plan nodes. Export it as Substrait, with an
> outcome that is either a plan or a reason there is none, never a
> half-converted one.
> Relations are checked before conversion: Impala's own relations are
> unambiguously outside Substrait, and listing them is more use than the
> exception Isthmus raises on the first one it meets. Expressions are left to
> Isthmus rather than to a list of what it refuses, which would drift and would
> also be wrong, since Impala's aggregate classes convert while its scalar
> functions do not.
> What Isthmus cannot resolve on its own is taught to it. Isthmus looks a call
> up in a map keyed on the operator and Calcite compares operators by class,
> name and kind, so an Impala operator meaning exactly what Substrait's does
> still misses. Signatures cover abs, coalesce, concat, length, floor and ceil
> with their d-prefixed spellings, substr and round, the window functions lag,
> first_value and last_value, and ndv. Three need more than a name: IF and cast
> arrive as operators carrying SqlKind.OTHER and get converters of their own,
> and substring, round and a datetime plus need their operands adjusted before
> a variant matches.
> What stays refused says why, in the table and in a test, so that a gap does
> not read as an oversight: Substrait has no truncate and no one-argument
> round, its approx_count_distinct takes one operand so a precision cannot be
> dropped quietly, GROUPING has no Substrait function at all, and SINGLE_VALUE
> would have to become any_value, which asserts nothing where Impala runs a
> cardinality check.
> This issue covers what was first filed as IMPALA-15297 through IMPALA-15302,
> IMPALA-15305 and IMPALA-15307, which were folded into it before review.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
