Aleksandr Efimov has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24750
Change subject: IMPALA-15304: Check the Substrait export reads back unchanged ...................................................................... IMPALA-15304: Check the Substrait export reads back unchanged The tests so far check what the exporter writes. None of them check what a consumer reads back, and a plan can serialize cleanly and still mean something else on the way through. substrait-java#1159 is exactly that, a multi-set aggregate whose columns came back in another order, and it was found from the reading side rather than the writing one. Read the exported plan back into Calcite with Isthmus and check it still says the same: the same relations in the same order, the same column types, and the same output names. Ten shapes, one per relation the seam can produce. Field names on the relations are not compared, and should not be. Substrait carries output names on the plan root rather than on the relations, so Calcite invents its own on the way back. The root names are what a consumer reads, and those are checked. One difference is pinned rather than worked around. Impala's string is a varchar of maximum width and the exported plan says so, but reading it back gives a varchar of 65536: SubstraitTypeSystem leaves RelDataTypeSystemImpl.getMaxPrecision alone, so Calcite caps it. What this patch writes is not the lossy part, the reading is, so it belongs in a test that names it rather than in a workaround here. Upstream issue that this shape of test is for: https://github.com/substrait-io/substrait-java/issues/1159 Testing, in a local cluster on Linux, the five classes in one run, 38 tests, 0 failures, 0 errors: - SubstraitRoundTripTest: 2 tests, the ten shapes and the varchar - SubstraitTpcdsExportTest: 1 test over 110 queries - SubstraitPlanExporterTest: 20 tests - CalciteOptimizerTest and CalciteSubstraitCompatibilityTest, neither of which this patch touches: 11 and 4 tests - git diff --check Change-Id: I663cd956938cc0111ea048386fa1921c7396216a Assisted-by: Claude Opus 5 (Anthropic) --- A java/calcite-planner/src/test/java/org/apache/impala/calcite/service/SubstraitRoundTripTest.java 1 file changed, 195 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/24750/1 -- To view, visit http://gerrit.cloudera.org:8080/24750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I663cd956938cc0111ea048386fa1921c7396216a Gerrit-Change-Number: 24750 Gerrit-PatchSet: 1 Gerrit-Owner: Aleksandr Efimov <[email protected]>
