LuciferYang opened a new pull request, #12981:
URL: https://github.com/apache/gluten/pull/12981
## What changes are proposed in this pull request?
Spark 3.3 support was removed in #12902, which left
`SparkVersionUtil.eqSpark33` constant-false on all four supported versions.
Every branch it guarded was dead. This removes those branches, and the same for
the test-side checks that are now constant.
In production code: `eqSpark33` and its three use sites are gone;
`AlignExpandOutputTypes` and its suite are deleted, since `VeloxRuleApi` only
registered the rule when `eqSpark33` held; `SparkTaskUtil` loses the
11-argument `TaskContextImpl` branch, which could not have succeeded on 3.4+
anyway because the only declared constructor takes 12; and `VeloxListenerApi`
loses the `startsWith("3.3")` post-rule registration.
Two reflection-based checks become direct type references, both of which
existed only because the type was absent on 3.3. `ExpressionConverter` matched
`RegExpInStr` through `getClass.getSimpleName` and now matches the type,
reading `subject` and `regexp` instead of `first` and `second`.
`SparkPlanUtil.isPlannedV1Write` reached `V1WriteCommand` through
`Utils.classForName` and now uses `isInstanceOf`, which also means a rename in
a future Spark breaks the build instead of silently returning false.
On the test side, five cases were pinned to 3.3 through
`testWithSpecifiedSparkVersion` / `testWithMaxSparkVersion`, so after the 3.3
removal they registered on no version at all. Four of them are `ignore` now:
their comments say they started failing on Spark 3.5 after the ClickHouse 25.12
rebase, so the version pin was standing in for an ignore. The fifth, `Native
writer should keep the same compression codec if hive.exec.compress.output is
true`, is deleted, because the code it covered was Gluten's own
`HiveFileFormat` override in `shims/spark32` and `shims/spark33` and no copy of
that file remains in the tree.
`testSparkVersionLE33` always took its `ignore` branch once 3.3 went, so it
is renamed to `ignoreSpark33OnlyCase` and its docstring records why its 13 call
sites stay registered as ignored. Eleven `if (spark33)` blocks had no `else`,
so they asserted nothing on a supported version; eight of them wrapped the
whole check of a TPC-H query and collapse to `check(N)`. Twelve
`isSparkVersionGE("3.4")` checks are now always true, and
`pruningTimeValueSpark` is now the constant 0. Two local `checkOperatorCount`
helpers had their whole body inside `if (spark33)`; they keep their call sites
and expected counts behind a TODO rather than losing them, so the enclosing CSE
tests still run and still assert nothing about the plan, as before.
One change adds an assertion rather than removing something.
`VeloxParquetWriteForHiveSuite` compares the written Parquet codec against
`zstd` in two places, and the deleted 3.3-only case was the only one that
wrapped the comparison in `assert`. The surviving case left it as a bare
expression whose value was discarded, so it now asserts.
## How was this patch tested?
Compile only, no suites were run. Everything below is Scala 2.13, because
the Spark 2.12 artifacts this machine resolves are patched and do not match
vanilla.
| profile | what was built |
|-|-|
| 3.5 | `gluten-core`, `gluten-substrait` incl. test sources, velox,
spark-ut, iceberg, clickhouse, kafka, delta |
| 3.4 | `gluten-core`, `gluten-substrait` incl. test sources, shim |
| 4.0 / 4.1 | the above plus arrow and velox |
`spotless:check` is clean on 3.5.
Not covered locally: no test suite ran, since the ClickHouse suites need
`libch.so` and the velox suites need the native library. That includes the
assertion added to `VeloxParquetWriteForHiveSuite`, which arms both iterations
of its `Seq(true, false)` loop, so CI is the first place it gets exercised on
either writer.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
