LuciferYang opened a new pull request, #13005:
URL: https://github.com/apache/gluten/pull/13005
## What changes are proposed in this pull request?
Residue from #12902 that guards on Spark 3.3 or 3.2, in code that no
supported version reaches. Most of it is in `backends-bolt`, because #12454
branched before #12954 and merged between #12954 and #12981, so it carried the
old patterns back in after the first cleanup and before the second one, which
touched no bolt file.
Never registered on any supported version:
`BoltParquetWriteForHiveSuite.scala:409` used `testWithMaxSparkVersion(...,
"3.3")`, so the case ran nowhere and was not reported as skipped either, since
the helper just declines to call `test()`. #12981 deleted the same case from
the velox copy of that suite, so this follows rather than decides.
Branches that cannot be taken: five `isSparkVersionGE("3.4")` guards in that
same file, `ArithmeticAnsiValidateSuite.scala:64`,
`MiscOperatorSuite.scala:962` in bolt on `startsWith("3.2")`, and the dead
`startsWith("3.3.")` half of a disjunction in `VeloxHashJoinSuite.scala:120`,
where the `startsWith("3.4.")` half stays because it still varies.
Helpers and settings with no callers: `ignoreWithSpecifiedSparkVersion` and
`testWithRangeSparkVersion` in `GlutenQueryTest.scala`, and
`BackendSettingsApi.rescaleDecimalArithmetic` with its two overrides. That last
one is named after the decimal rescale path #12991 removes, and it had no
reader before that either.
An obsolete workaround: `CudfNodeValidationRule.scala:35` walked the plan
with `find { ... }.isDefined` under a comment saying `Spark3.2 does not have
exists`. `TreeNode.exists` is on every supported version and already used in
`Validators.scala` and `ImplementJoinAggregate.scala`.
Files nothing compiles: `gluten-ut/spark33/` holds two bolt test-settings
files and no `pom.xml`, and
`backends-bolt/src/test/resources/tpch-approved-plan/*/spark33/` holds 84
approved plans, with another 84 under `*/spark32/`. `gluten-ut/pom.xml` lists
only `spark34/35/40/41`, so none of the Scala is built, though
`scala_code_format.yml` triggers on `gluten-ut/**` and still formats it. That
is the bulk of the diffstat.
Worth flagging separately, because it is a bolt question rather than
something to settle here: the only reader of `tpch-approved-plan` anywhere is
`VeloxTPCHSuite.scala:105`, and it resolves the path relative to the running
module. There is no bolt TPC-H suite, so the `spark34` and `spark35` copies
under `backends-bolt` have no reader either. They stay, since a bolt suite may
be the intent. Also `BoltParquetWriteForHiveSuite.scala:285` evaluates
`"zstd".equalsIgnoreCase(column.getCodec.toString)` and discards it: the
missing `assert` is pre-existing, but with the 3.3-only case gone nothing in
that file checks the written codec any more.
Four comments that had become wrong rather than merely old:
`SparkDataSourceRDDPartition` said it exists to stay compatible with 3.3 and
earlier, when what actually distinguishes it from Spark's class is that it
holds the several input partitions Gluten groups into one split; the four
copies of `WritableColumnVectorShim` named a 3.3 abstract-method addition as
the reason the file is per-version, when the copies now split {3.4, 3.5}
against {4.0, 4.1} over something else. Also `// Since spark 3.3.0` in two
backends, `// For spark 3.2.` in one, a commented-out `enableSuite` under `//
following UT is removed in spark3.3.1`, a duplicated `"3.5", "3.5"` argument in
two cache suites left over from a merge fix in #9297, and `All the spark
version is supported` in `VeloxIceberg.md`.
Deliberately left alone: the `// new added in spark-3.3 and need fix later,
random failure may caused by memory free` notes in the gluten-ut settings and
the `For Spark 3.3 + Delta 2.3` notes in the clickhouse tests. Only the version
prefix has aged; the rest of each sentence is still the reason the exclusion or
the expected file count is there, and rewording someone else's note across
sixteen files buys nothing.
## How was this patch tested?
`clean test-compile` on Spark 3.4, 3.5, 4.0 and 4.1 with `-Pbackends-velox
-Pspark-ut -Piceberg -Pdelta`, Scala 2.13, and the 3.5 run adds
`-Pbackends-clickhouse`, that being the only profile clickhouse builds on.
`spotless:apply` produced no changes beyond the edits.
The bolt files needed a detour, since bolt does not compile on main right
now: I stacked #12999 on top of this branch, ran `test-compile -Pbackends-bolt
-Pspark-ut` on 3.4 and 3.5, then dropped it, so what is pushed here is this
change alone.
No suite was run. Every collapsed branch keeps the arm a supported version
already took, and the deleted files are not on any compile or test path. The
bolt paths cannot be run here in any case: bolt's native library needs Linux
plus GCC 10-12 or Clang 16 per `docs/bolt-quick-start.md`.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5
Related issue: #13003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
