Aleksandr Efimov created IMPALA-15305:
-----------------------------------------
Summary: Calcite planner: export NDV as a Substrait
approx_count_distinct
Key: IMPALA-15305
URL: https://issues.apache.org/jira/browse/IMPALA-15305
Project: IMPALA
Issue Type: Sub-task
Components: Frontend
Reporter: Aleksandr Efimov
NDV was left out of the mapping table added by IMPALA-15297 on the grounds that
approx_count_distinct is the obvious counterpart but the rename would assert
something about the estimator that a name alone does not back.
Checking rather than assuming settles it. Substrait says of
approx_count_distinct that it calculates the approximate number of distinct
values "using HyperLogLog". Impala's ndv is HyperLogLog too, sized by
ComputeHllLengthFromScale. Both name the same algorithm, so the mapping asserts
nothing that is not already true of either side.
Map it, and wire an aggregate converter to carry it: the provider had scalar
and window converters until now.
The two-argument ndv, which sets the precision, needs nothing to stay refused.
Substrait has one variant of approx_count_distinct and it takes one operand, so
a call carrying a scale matches nothing and the export says so. There is no way
to drop the precision without noticing, and nothing in the mapping arranges
that.
No TPC-DS query is affected: ndv is an Impala function rather than standard
SQL, so the corpus stays at 102 of 110. It is common in real Impala queries,
which is the reason to carry it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
