akshaytayal opened a new pull request, #13518: URL: https://github.com/apache/gluten/pull/13518
## What changes are proposed in this pull request? Part of #13517. Fixes Spark 4.3 API changes that affect code shared by all Spark profiles, like #13020 did for Spark 4.2. Every Spark 4.3 path is behind `SparkVersionUtil.gteSpark43` or reflection, so this PR needs no Spark 4.3 jars and Spark 3.4–4.2 behavior is unchanged. | Spark change | Fix | |---|---| | SPARK-56981 adds `getTimestampNTZNanos` / `getTimestampLTZNanos` to `SpecializedGetters` | Stubs in `SpecializedGettersSparkCompatible` / `InternalRowSparkCompatible`, like `getVariant` | | SPARK-58185 moves `shuffleBlockResolver` to the new `private[spark] BlockingShuffleManager` | `ShuffleManagerRouter` / `GlutenShuffleManager` no longer mark it `override`; the router looks up the delegate's resolver reflectively. `GlutenShuffleManager` can't be used on Spark 4.3 yet (TODO). `ColumnarShuffleManager` is not affected | | SPARK-58192 makes task CPUs a `BigDecimal` | `GlutenAutoAdjustStageResourceProfile` uses exact floor division; `SparkTaskUtil` passes the type the constructor declares | | SPARK-58084 adds `countLocalSort` to `SimpleCostEvaluator` | `GlutenCostEvaluator` passes it on Spark 4.3+ (new `SparkVersionUtil.gteSpark43`) | | SPARK-58119 passes a `PythonWorkerHandle` instead of a pid to `newReaderIterator` | `BasePythonRunnerShim.WorkerHandle` type alias, used by `ColumnarArrowPythonRunner` | | SPARK-57851 lets a `FileSourceScanExec` report `SinglePartition` (single-task execution, off by default) | Such scans fall back, since the native scan doesn't coalesce its splits (`SparkShims.isSingleTaskExecutionScan`) | `MiscOperatorSuite` and `GlutenBloomFilterFallbackSuite` ignore their tests on Spark 4.2+ instead of 4.2 only (GLUTEN-13179), since Spark 4.3 keeps the same function identifier requirement. The shim layer and build profile (#13517 B) and the unit test module (#13517 C) follow in separate PRs. ## How was this patch tested? Existing CI for Spark 3.4–4.2. Built locally against Spark 4.3.0 together with the Spark 4.3 shim and UT PRs. The same changes were validated earlier against Spark 4.3.0-RC2 in a private CI run, with all Spark 3.4–4.2 and Spark 4.3 UT jobs passing. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot CLI (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
