yihua opened a new pull request, #20551:
URL: https://github.com/apache/hudi/pull/20551

   ### Describe the issue this Pull Request addresses
   
   Closes #20550
   
   Part of #16053.
   
   **Before merge:** the base image `flink1200hive313spark430scala213` has to 
be published to Docker Hub by a committer, using 
`packaging/bundle-validation/base/build_flink1200hive313spark430scala213.sh`. 
Until then `validate-bundle-spark4` fails on 4.3, which also fails the required 
`Java CI Engines result` check.
   
   ### Summary and Changelog
   
   Hudi can now be built and run on Spark 4.3.0 (Scala 2.13, JDK 17) with 
`-Dspark4.3`. The new `hudi-spark4.3.x` module is copied from 
`hudi-spark4.2.x`, following #18621, and the wiring mirrors that PR.
   
   The changes that go beyond the copy:
   
   - **Spark API changes:** adapts to the new `ParserInterface` methods 
(through a per-version source root in `hudi-spark`, since the new types only 
exist on 4.3), the `AlterColumns` and `AlterTableChangeColumnCommand` arity 
changes, the new nanos row getters, the removal of `ParserUtils.validate`, and 
the `ArrowUtils` signature used by the Vortex writer.
   - **Dependency order in `hudi-spark4.3.x/pom.xml`:** `hudi-spark-common` is 
declared before `spark-sql`. Spark 4.3's `AvroSchemaHelper` constructor takes 6 
args and the `AvroUtils` vendored by Hudi takes 5, so compiling against Spark's 
version throws `NoSuchMethodError` at runtime. A bump to Spark 4.2.1 would need 
the same ordering.
   - **Tests:** Spark 4.3 compares both analyzers' plans when `spark.testing` 
is set, and Hudi's rules only run in the fixed-point analyzer, so the root pom 
turns off `spark.sql.analyzer.singlePassResolver.dualRunWithLegacy` for 
surefire and scalatest.
   - **CI:** the JDK 17 lanes move from 4.2 to 4.3, with 4.2 parked as 
comments. Broader matrix coverage will come in separate PRs.
   
   ### Impact
   
   Adds Spark 4.3 as a supported engine. `-Dspark4.3` produces no native 
bundle, because Comet publishes no artifact for 4.3.
   
   Known gaps on 4.3:
   
   - Multi-column and nested `COMMENT ON` 
(`AlterTableChangeColumnCommentsCommand`) isn't rewritten by Hudi's rules.
   - Hudi's rules don't run under the single-pass resolver. It's off by default.
   - `TemporalIdentifier` reads such as `spark.read.table("t@v1")` are untested.
   - The forked Parquet readers aren't re-synced for nanos timestamps, and the 
row wrappers only delegate.
   - Unrelocated jars need `hudi-spark-common` ahead of Spark on the classpath, 
or the Avro forks throw `NoSuchMethodError`. The bundles aren't affected, but 
`hudi-integ-test` declares `spark-sql` first, and no CI lane runs it on 4.3.
   
   ### Risk Level
   
   Low. Most of the change is new code that only runs on 4.3. The shared 
changes are version-gated or compile-time seams, and the main compile passes on 
Spark 3.3 through 4.2.
   
   Local runs on 4.3: 508 of 513 DDL and DML tests passed, and the affected 
suites passed on rerun (four of the five failures were local port collisions, 
the other a heartbeat expiry during a JVM stall). A bundle smoke test (COW and 
MOR writes, schema evolution, time travel, on real 4.3 jars) also passed. 
Functional suites and most `hudi-utilities` tests are left to this PR's CI. 4.2 
loses its JDK 17 CI lanes until the matrix is expanded.
   
   ### Documentation Update
   
   `README.md` and `hudi-spark-datasource/README.md` list the new build 
profile. The website needs an update once 4.3 ships in a release.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to