[
https://issues.apache.org/jira/browse/IMPALA-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov updated IMPALA-15297:
--------------------------------------
Description:
IMPALA-15296 added a test-scoped exporter for the pre-conversion logical plan.
Twenty of the twenty-four query shapes the direct harness covers export end to
end. The rest stop on an operator Isthmus cannot resolve.
Isthmus resolves a call by looking the operator up in a map keyed on the
operator itself, and Calcite compares operators by class, name and kind. Impala
substitutes its own classes for a good number of functions, so an operator that
means exactly what Substrait's does still misses. Isthmus takes additional
signatures for this: they say which Substrait function an Impala operator
stands for and leave the operand matching alone.
Add them for the operators that have a counterpart. For scalars that is
coalesce, concat, length, and floor and ceil with their d-prefixed spellings;
Impala reaches those by three routes, the USE_IMPALA_OPERATOR set in
ImpalaOperatorTable, builtins Calcite has no operator for, and the replacements
in ImpalaCustomOperatorTable, and all three resolve by name. For windows it is
lag, first_value and last_value, resolved through a converter of their own that
first tries to normalise the aggregate by SqlKind; Impala's window operators
carry SqlKind.OTHER, so normalisation passes them through and the lookup lands
on the operator itself. lead needs nothing, since USE_IMPALA_OPERATOR lists lag
but not lead.
Together that takes the corpus from twenty to twenty-three.
What is left needs a different mechanism rather than another name. Substrait
models a conditional as the IfThen expression rather than as a function, so
Impala's IF has no name to map to, and it is the one corpus shape still not
exporting. Substrait's round takes its digit count as i32 where Impala types
the literal as i8, so the name lines up but the operands do not, and Substrait
has no truncate at all. NDV is left out on other grounds: approx_count_distinct
is the obvious counterpart, but that is a rename carrying a claim about the
estimator rather than a matter of operator class.
was:
IMPALA-15296 added a test-scoped exporter for the pre-conversion logical plan.
Twenty of the twenty-four query shapes the direct harness covers export end to
end. The rest stop on an operator Isthmus cannot resolve.
Isthmus resolves a scalar call by looking the operator up in a map keyed on the
operator itself, and Calcite compares operators by class, name and kind. Impala
substitutes its own classes for a good number of functions, so an operator that
means exactly what Substrait's does still misses. Isthmus takes additional
signatures for this: they say which Substrait function an Impala operator
stands for and leave the operand matching alone.
Add the signatures for the operators that have a Substrait counterpart:
coalesce, concat, length, and floor and ceil with their d-prefixed spellings.
Impala reaches these by three separate routes, the USE_IMPALA_OPERATOR set in
ImpalaOperatorTable, builtins that Calcite has no operator for, and the
replacements in ImpalaCustomOperatorTable. All three resolve by name, so one
table covers them.
That takes the harness corpus from twenty to twenty-two. What is left needs a
different mechanism rather than another name. Substrait models a conditional as
the IfThen expression rather than as a function, so Impala's IF has no name to
map to, and lag is a window function resolved by a different converter.
Substrait's round takes its digit count as i32 where Impala types the literal
as i8, so the name lines up but the operands do not, and Substrait has no
truncate at all.
> Calcite planner: map Impala operators onto Substrait functions
> --------------------------------------------------------------
>
> Key: IMPALA-15297
> URL: https://issues.apache.org/jira/browse/IMPALA-15297
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> IMPALA-15296 added a test-scoped exporter for the pre-conversion logical
> plan. Twenty of the twenty-four query shapes the direct harness covers export
> end to end. The rest stop on an operator Isthmus cannot resolve.
> Isthmus resolves a call by looking the operator up in a map keyed on the
> operator itself, and Calcite compares operators by class, name and kind.
> Impala substitutes its own classes for a good number of functions, so an
> operator that means exactly what Substrait's does still misses. Isthmus takes
> additional signatures for this: they say which Substrait function an Impala
> operator stands for and leave the operand matching alone.
> Add them for the operators that have a counterpart. For scalars that is
> coalesce, concat, length, and floor and ceil with their d-prefixed spellings;
> Impala reaches those by three routes, the USE_IMPALA_OPERATOR set in
> ImpalaOperatorTable, builtins Calcite has no operator for, and the
> replacements in ImpalaCustomOperatorTable, and all three resolve by name. For
> windows it is lag, first_value and last_value, resolved through a converter
> of their own that first tries to normalise the aggregate by SqlKind; Impala's
> window operators carry SqlKind.OTHER, so normalisation passes them through
> and the lookup lands on the operator itself. lead needs nothing, since
> USE_IMPALA_OPERATOR lists lag but not lead.
> Together that takes the corpus from twenty to twenty-three.
> What is left needs a different mechanism rather than another name. Substrait
> models a conditional as the IfThen expression rather than as a function, so
> Impala's IF has no name to map to, and it is the one corpus shape still not
> exporting. Substrait's round takes its digit count as i32 where Impala types
> the literal as i8, so the name lines up but the operands do not, and
> Substrait has no truncate at all. NDV is left out on other grounds:
> approx_count_distinct is the obvious counterpart, but that is a rename
> carrying a claim about the estimator rather than a matter of operator class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
