LuciferYang opened a new pull request, #12532:
URL: https://github.com/apache/gluten/pull/12532
### What changes were proposed in this pull request?
Spark 3.2 was dropped by #11351 / #11687 / #11731 / #11887; the `spark32`
protected def in `GlutenClickHouseWholeStageTransformerSuite` was defined as
`sparkVersion.equals("3.2")` and has been dead since. This PR removes the
definition and prunes every reachable `if (spark32) ...` / `if (!spark32) ...`
/ `${if (spark32) ... else ...}` branch to keep only the Spark 3.3+ path.
**Test-code changes** (all in `backends-clickhouse/src/test/`):
- `GlutenClickHouseWholeStageTransformerSuite.scala`: drop the `protected
def spark32` definition (always false).
- `GlutenClickHouseTPCHBucketSuite.scala`: `hasSortByCol = !spark32`
collapses to `true` on all supported Sparks; version-gated `if (spark32) ...`
branches removed.
- `GlutenClickHouseTPCHParquetBucketSuite.scala`,
`GlutenClickHouseDeltaParquetWriteSuite.scala`,
`GlutenClickHouseMergeTreeWriteSuite.scala`,
`GlutenClickHouseMergeTreeOptimizeSuite.scala`,
`GlutenClickHouseMergeTreeWriteOnHDFSSuite.scala`,
`GlutenClickHouseMergeTreeWriteOnHDFSWithRocksDBMetaSuite.scala`,
`GlutenClickHouseMergeTreeWriteOnS3Suite.scala`,
`GlutenClickHouseMergeTreePathBasedWriteSuite.scala`: every reachable `if
(spark32) ... else ...` block, `if (!spark32) ...` guard, and inline `${if
(spark32) "" else "SORTED BY (...)"}` interpolation is reduced to the Spark
3.3+ path (always emit `SORTED BY`).
- `GlutenClickHouseTPCDSParquetAQESuite.scala`,
`GlutenClickHouseTPCDSParquetColumnarShuffleAQESuite.scala`: comments narrowed
from "On Spark 3.2, ... on Spark 3.3, ..." to describe only the surviving Spark
3.3+ shape.
- `hive/GlutenClickHouseNativeWriteTableSuite.scala`: drop stale `// spark
3.2 without orc or parquet suffix` comment.
**Main-code change** (one file):
- `RowToCHNativeColumnarExec.scala`: drop the `// For spark 3.2.` comment
above `withNewChildInternal`. The override is required by `TreeNode`'s API on
every Spark version currently supported by Gluten, not a Spark 3.2-only quirk.
Mirrors the same cleanup for `RowToVeloxColumnarExec` included in #12525.
**Explicitly kept for a separate follow-up PR** (real refactor, not comment
fix):
- `backends-clickhouse/.../ExtendedColumnPruning.scala:66-72` — the local
`getAttributeToExtractValues` re-implementation exists because Spark 3.2's
upstream signature was 2-arg. On 3.3+ it is 3-arg; the local copy could be
replaced with a delegate.
- `backends-clickhouse/.../CHColumnarWrite.scala:157` — the `bucketSpec`
reflection was needed for Spark 3.2, may be replaceable with direct access on
3.3+.
- `CustomSum.scala:28` — historical provenance of a copied file, not a
version gate; keep as-is.
### How was this patch tested?
- `mvn -pl backends-clickhouse -am install
-Pspark-3.3,backends-clickhouse,delta`: SUCCESS
- `mvn -pl backends-clickhouse scalastyle:check spotless:check
-Pspark-3.3,backends-clickhouse,delta`: SUCCESS
- The pruned branches were dead code on every currently supported Spark
profile, so runtime behavior is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
