Aleksandr Efimov created IMPALA-15306:
-----------------------------------------
Summary: Calcite planner: carry Calcite's estimates into the
Substrait plan
Key: IMPALA-15306
URL: https://issues.apache.org/jira/browse/IMPALA-15306
Project: IMPALA
Issue Type: Sub-task
Components: Frontend
Reporter: Aleksandr Efimov
A plan leaves the exporter saying nothing about how much data any relation is
expected to produce. Isthmus derives no statistics of its own, so a consumer
planning or costing the query has to guess. Substrait has a place for this on
every relation, and Calcite has already worked the numbers out by the time the
plan reaches the seam.
Carry them out. SubstraitRelVisitor.apply is public and every relation goes
through it, so a subclass can attach RelMetadataQuery's row count and average
row size to what it returns.
Nothing is asserted that Substrait does not already treat as approximate: its
own words for the field are "estimated statistics" and "the estimated row
count". That matters, because Impala's row count is a real one from COMPUTE
STATS when the table has been analysed and an estimate derived from file sizes
when it has not, and CalciteTable hands both over through the same getRowCount.
Writing either into a field declared as an estimate says nothing untrue;
writing them into a field that promised a fact would.
The estimates are attached to what reaches apply rather than to the Calcite
node they came from. Isthmus drops a projection that only renames, returning
its input instead, and rewrites an aggregate before the visitor sees it, so
binding to the returned relation keeps the numbers on the relation a consumer
will read.
Requires substrait-java 0.101.0: Rel.withHint arrived there, and 0.100.0 can
read a hint but has no type-agnostic way to set one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
