Hello Joe McDonnell, Steve Carlin, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24749
to look at the new patch set (#12).
Change subject: IMPALA-15303: Cover a real workload and the reading side
......................................................................
IMPALA-15303: Cover a real workload and the reading side
The exporter is covered by query shapes written by hand, one construct
at a time, which says nothing about what a real workload does or about
what a consumer reads back. This adds both.
Run the exporter over the TPC-DS queries that carry Calcite goldens.
Every gap the exporter turned out to have was found this way rather than
by hand. The tables come from the schema template rather than from the
catalogue, so this runs wherever the frontend tests run and needs no
loaded warehouse; only column types matter for an export, which also
means the plans are not the ones a loaded warehouse would produce.
The expectation is a named list rather than a count: which queries do
not export and what stops each. A count would only say the number moved;
removing the abs mapping, for instance, fails the test with tpcds-q47,
q53, q57, q63 and q89 named against ABS. Today 95 of the 103 distinct
queries export, the eight that do not being outside this exporter, four
on SINGLE_VALUE and four on GROUPING (substrait-java#993). A query
leaving that list matters as much as one joining it, so both directions
are reported.
Read the exported plan back with Isthmus and check it still says the
same: the same relations in the same order, the same column types, the
same output names. A plan can serialize cleanly and still mean something
else on the way through -- substrait-java#1159, found from the reading
side rather than the writing one. Field names on the relations are not
compared, and should not be: Substrait carries output names on the plan
root, so Calcite invents its own coming back.
Testing, in a local cluster on Linux:
- SubstraitTpcdsExportTest: 1 test over the 103 distinct queries in 110
files, about eight seconds including the schema fixture
- SubstraitRoundTripTest: 2 tests, ten shapes and the varchar width that
substrait-java#1169 gives back, which is what pins 0.103.0
- SubstraitPlanExporterTest, CalciteOptimizerTest and
CalciteSubstraitCompatibilityTest, none of which this patch touches:
0 failures
Assisted-by: Claude Opus 5 (Anthropic)
Change-Id: If0f8921a4fd3bde41e227f7c0e986e82cf0d3ffa
---
A
java/calcite-planner/src/test/java/org/apache/impala/calcite/service/SubstraitRoundTripTest.java
A
java/calcite-planner/src/test/java/org/apache/impala/calcite/service/SubstraitTpcdsExportTest.java
2 files changed, 434 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24749/12
--
To view, visit http://gerrit.cloudera.org:8080/24749
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If0f8921a4fd3bde41e227f7c0e986e82cf0d3ffa
Gerrit-Change-Number: 24749
Gerrit-PatchSet: 12
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
