zhouyuan opened a new pull request, #13019:
URL: https://github.com/apache/gluten/pull/13019
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
Prototype, spark35 + Velox only, to see how much of gluten-ut is boilerplate
that does not need to be written by hand once per Spark version.
Two data files replace the hand-written code:
src/test/resources/gluten-ut/suites.txt the suite catalog
src/test/resources/gluten-ut/settings-velox.txt what runs, per backend
tools/scripts/gluten-ut/ut_suites.py generates the `class GlutenXSuite
extends XSuite with GlutenSQLTestsBaseTrait {}` subclasses from the catalog at
generate-test-sources, into target/, so there is nothing to commit or review.
Generated code fully qualifies the base class, so a bad entry is a compile
error rather than a silently mis-wired suite -- which is how the two subclasses
that live in a different package than the suite they extend
(GlutenValidateRequirementsSuite, GlutenStreamingQuerySuite) were found.
DataFileBackendTestSettings reads the settings file at run time and drives
the existing BackendTestSettings DSL, so include/exclude/by-prefix semantics
are unchanged. It resolves every enabled suite with Class.forName at load time,
so a typo fails fast instead of quietly skipping a suite. This is the shape the
Delta suite already uses
(.github/workflows/util/delta-spark-ut/known-failures.txt): expected failures
as reviewable data, which a version bump can diff and CI can gate against a
baseline.
spark35/src/test/scala: 269 -> 117 files. VeloxTestSettings.scala: 988 -> 27
lines. 5 subclasses stay hand-written because they share a file with other code.
Also GlutenVanillaPlanView, for the other half of the cost: tests copied
into gluten-ut only because they assert on the operator type, and after offload
a SortExec is a SortExecTransformer. It overrides the AdaptiveSparkPlanHelper
helpers Spark's tests use (collect, find, collectFirst, collectWithSubqueries)
so a predicate matches an offloaded operator wherever it would match the
vanilla one, leaving `collect(plan) { case _: SortExec => }` working
unmodified. It is additive: assertions on *Transformer types still match,
unrelated types still do not. Only same-operator pairs belong in the mapping --
a test wanting SortMergeJoinExec where Gluten plans a shuffled hash join is
asserting something genuinely no longer true. Covers
Sort/Project/Filter/Window; scans, joins and aggregates are next.
Verification:
- ut_suites.py verify: the catalog reproduces all 165 subclasses.
- ut_suites.py extract cross-checks that every DSL call site was consumed.
- VeloxSettingsMigrationSuite: the old and new settings enable the same
suites and make the same run/skip decision for every test name they mention.
This caught a real extractor bug -- names written as "SPARK-19471: ..." + "
before using it" were being split in two.
- GlutenVanillaPlanViewInterceptionSuite: the helper overrides do redirect
plan assertions (stubbed mapping, so no backend needed).
- All 282 sources in spark35 compile with GlutenVanillaPlanView mixed into
GlutenSQLTestsBaseTrait, i.e. no conflicting-member breakage.
Not yet verified: GlutenVanillaPlanViewSuite (the real operator mapping)
needs a current backend jar, and nothing here has run against a live Gluten
plan, which needs the native library. Build-time generation needs python3 on
PATH.
VeloxTestSettingsLegacy.scala and VeloxSettingsMigrationSuite.scala exist
only to prove equivalence; delete both if this lands.
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after
updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the
corresponding issue
when the PR is merged.
-->
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
## Was this patch authored or co-authored using generative AI tooling?
Claude code Opus
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
