Aleksandr Efimov has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24749
Change subject: IMPALA-15303: Cover the TPC-DS corpus in the Substrait export tests ...................................................................... IMPALA-15303: Cover the TPC-DS corpus in the Substrait export tests The exporter is covered by query shapes written by hand, one construct at a time. Every gap it turned out to have was found by running a real workload instead, and the numbers those runs produced have been quoted in IMPALA-15297 and after without anything in the tree that reproduces them. Run the exporter over the TPC-DS queries that carry Calcite goldens, as a test. The tables come from the schema template rather than from the catalogue, so this runs wherever the frontend tests run and needs no loaded warehouse. Only column types matter for an export, so no data or statistics are involved, which also means the plans here are not the ones a loaded warehouse would produce. The expectation is a named list rather than a count: which queries do not export and what stops each. A count would only say that the number moved. Removing the abs mapping, for instance, fails the test with tpcds-q47, q53, q57, q63 and q89 named against ABS. Today 102 of 110 export. The eight that do not are outside this exporter: four stop on SINGLE_VALUE, which Calcite uses to guard a scalar subquery and Impala turns into a runtime cardinality check that Substrait has no way to express, and four on GROUPING, which is substrait-java#993. A query leaving that list matters as much as one joining it, so the test reports both directions. Upstream issue: https://github.com/substrait-io/substrait-java/issues/993 Testing, in a local cluster on Linux, the four classes in one run: - SubstraitTpcdsExportTest: 1 test over 110 queries, 0 failures. Takes about nine seconds, the schema fixture included - SubstraitPlanExporterTest: 20 tests, 0 failures, 0 errors - CalciteOptimizerTest and CalciteSubstraitCompatibilityTest, neither of which this patch touches: 11 and 4 tests, 0 failures - Dropping a mapping fails the added test with the affected queries named, so it is not vacuous - git diff --check Change-Id: If0f8921a4fd3bde41e227f7c0e986e82cf0d3ffa Assisted-by: Claude Opus 5 (Anthropic) --- A java/calcite-planner/src/test/java/org/apache/impala/calcite/service/SubstraitTpcdsExportTest.java 1 file changed, 228 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24749/1 -- To view, visit http://gerrit.cloudera.org:8080/24749 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If0f8921a4fd3bde41e227f7c0e986e82cf0d3ffa Gerrit-Change-Number: 24749 Gerrit-PatchSet: 1 Gerrit-Owner: Aleksandr Efimov <[email protected]>
