akshaytayal opened a new pull request, #13519:
URL: https://github.com/apache/gluten/pull/13519
## What changes are proposed in this pull request?
Part of #13517. Adds the Spark 4.3 shim layer and build profile for the
Velox backend, like #13126 for Spark 4.2.
**Depends on #13518.** Until #13518 merges, this branch includes its commit;
this PR's own changes are the last 3 commits. Following #11352 (example
#11347), they keep the history of the moved files and should be merged with
**rebase**, not squash:
1. `Spark: Move 4.2 as 4.3`: `git mv shims/spark42 shims/spark43` (renames
only).
2. `Spark: Copy back 4.3 as 4.2`: restores `shims/spark42` unchanged.
3. `Spark: Initial support for 4.3.0`: the Spark 4.3 changes:
- `pom.xml`: `spark-4.3` profile (`spark.version` 4.3.0). Spark 4.3.0 has
the same dependency versions as Spark 4.2.0, so the profile keeps the
`spark-4.2` overrides. `spark-4.3` is added to the required Spark profile list.
- `shims/spark43`: `Spark43Shims`, `SparkShimProvider` and the shim test
suites renamed from 4.2, plus:
- `ArrowColumnarArray` implements `getTimestampNTZNanos` /
`getTimestampLTZNanos` (SPARK-56981).
- `FileSourceScanExecShim.markedForSingleTaskExecution` is false
(SPARK-57851), and `Spark43Shims.isSingleTaskExecutionScan` reports the
original scan's `useSingleTaskExecution`, so such scans fall back.
- `BasePythonRunnerShim.WorkerHandle` is `Option[PythonWorkerHandle]`
(SPARK-58119).
- Wiring: `shims/pom.xml`, `package/pom.xml`, `tools/gluten-it/pom.xml`,
`install-spark-resources.sh` (4.3), `velox_backend_x86.yml` (`shims43` change
detection and a build-only `build-test-spark43` job), Dockerfiles and dev
scripts.
No Delta, Iceberg, Hudi or Paimon release supports Spark 4.3 yet. Their
properties in the `spark-4.3` profile are placeholders inherited from
`spark-4.2`, and those profiles are not enabled for Spark 4.3 in CI.
`gluten-ut/spark43` follows in a separate PR (#13517 C).
## How was this patch tested?
`build-test-spark43` builds Gluten with `-Pspark-4.3`. Built locally against
Spark 4.3.0, including all test sources, together with the Spark 4.3 UT PR. The
same changes were validated earlier against Spark 4.3.0-RC2 in a private CI run.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
