[
https://issues.apache.org/jira/browse/IMPALA-15303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov updated IMPALA-15303:
--------------------------------------
Description:
The exporter added by IMPALA-15296 is covered by query shapes written by hand,
one construct at a time. Two things that says nothing about: what a real
workload does, and what a consumer reads back.
Run the exporter over the TPC-DS queries that carry Calcite goldens. Every gap
the exporter turned out to have was found that way rather than by hand. The
tables are built from the schema template rather than taken from the catalogue,
so this runs wherever the frontend tests run. The expectation is a named list
rather than a count: which queries do not export and what stops each, so a
change names the query it moved. 95 of the 103 distinct queries export today;
the directory holds 110 files, seven of which are plan variants of a query
already in it.
Read the exported plan back with Isthmus and check it still says the same
relations, types and output names. A plan can serialize cleanly and still mean
something else on the way through, which is substrait-java#1159, found from the
reading side. Reading uses the provider that synthesises operators for
functions the stock signature tables do not name, which is what a consumer
reading arbitrary Substrait would use; the exporter's own provider is for
writing only.
One difference is pinned rather than worked around: Impala's string is a
varchar of maximum width and comes back capped at 65536. Reaching Substrait's
unbounded string through a UserTypeMapper does not work, because the mapped
type is dropped when a literal is rebuilt from the Calcite type name, which is
substrait-java#1170.
This issue also covers what was first filed as IMPALA-15304, folded into it
before review.
was:
The exporter is covered by query shapes written by hand, one construct at a
time. Every gap it turned out to have was found by running a real workload
instead, and the numbers those runs produced have been quoted in IMPALA-15297
and after without anything in the tree that reproduces them.
Run the exporter over the TPC-DS queries that carry Calcite goldens, as a test.
The tables come from the schema template rather than from the catalogue, so
this runs wherever the frontend tests run and needs no loaded warehouse. Only
column types matter for an export, so no data or statistics are involved, which
also means the plans are not the ones a loaded warehouse would produce.
The expectation is a named list rather than a count: which queries do not
export and what stops each. A count would only say that the number moved.
Removing the abs mapping, for instance, fails the test with tpcds-q47, q53,
q57, q63 and q89 named against ABS.
Today 102 of 110 export. The eight that do not are outside this exporter: four
stop on SINGLE_VALUE, which Calcite uses to guard a scalar subquery and Impala
turns into a runtime cardinality check that Substrait has no way to express,
and four on GROUPING, which is substrait-java#993. A query leaving that list
matters as much as one joining it, so the test reports both directions.
Summary: Calcite planner: cover a real workload and the reading side of
the Substrait export (was: Calcite planner: cover the TPC-DS corpus in the
Substrait export tests)
> Calcite planner: cover a real workload and the reading side of the Substrait
> export
> -----------------------------------------------------------------------------------
>
> Key: IMPALA-15303
> URL: https://issues.apache.org/jira/browse/IMPALA-15303
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> The exporter added by IMPALA-15296 is covered by query shapes written by
> hand, one construct at a time. Two things that says nothing about: what a
> real workload does, and what a consumer reads back.
> Run the exporter over the TPC-DS queries that carry Calcite goldens. Every
> gap the exporter turned out to have was found that way rather than by hand.
> The tables are built from the schema template rather than taken from the
> catalogue, so this runs wherever the frontend tests run. The expectation is a
> named list rather than a count: which queries do not export and what stops
> each, so a change names the query it moved. 95 of the 103 distinct queries
> export today; the directory holds 110 files, seven of which are plan variants
> of a query already in it.
> Read the exported plan back with Isthmus and check it still says the same
> relations, types and output names. A plan can serialize cleanly and still
> mean something else on the way through, which is substrait-java#1159, found
> from the reading side. Reading uses the provider that synthesises operators
> for functions the stock signature tables do not name, which is what a
> consumer reading arbitrary Substrait would use; the exporter's own provider
> is for writing only.
> One difference is pinned rather than worked around: Impala's string is a
> varchar of maximum width and comes back capped at 65536. Reaching Substrait's
> unbounded string through a UserTypeMapper does not work, because the mapped
> type is dropped when a literal is rebuilt from the Calcite type name, which
> is substrait-java#1170.
> This issue also covers what was first filed as IMPALA-15304, folded into it
> before review.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
