yihua opened a new pull request, #19674:
URL: https://github.com/apache/hudi/pull/19674
### Describe the issue this Pull Request addresses
closes #19673
### Summary and Changelog
Adds `packaging/hudi-native-spark-bundle`, the contents of
`hudi-spark-bundle` plus Apache
DataFusion Comet. Comet publishes one artifact per Spark minor version and
does not cover
everything Hudi builds against, so the module defaults to producing nothing
and each supported
Spark profile opts in:
| Spark profile | Comet at 1.0.0 | Bundle |
|---|---|---|
| 3.4 (Scala 2.12), 3.5 (2.12, 2.13), 4.0, 4.1 (2.13) | published | produced
|
| 3.3 (dropped after Comet 0.7.0), 4.2 (unreleased) | none | not produced |
A build for an uncovered Spark version succeeds and emits nothing, rather
than a jar named
"native" carrying no native code.
- New module and README. Its pom is `hudi-spark-bundle`'s with five changes:
artifact id, a shade
execution id, `org.apache.datafusion:*` in the shade artifact set, the
skip switch, and one
profile per supported Spark version.
- `pom.xml`: `comet.version` and the new module.
- `scripts/release/`: build, deploy and staged-bundle validation for the
five new coordinates.
- `packaging/bundle-validation/native_spark/`, `.github/workflows/bot.yml`:
a `validate-native-bundle`
job. It runs on the runner rather than in the bundle-validation image,
because that image is
Alpine based and Comet's `libcomet.so` is glibc-linked, so it cannot load
there.
- `LICENSE`, `NOTICE`: Comet, and the BSD-3-Clause Protocol Buffers it
redistributes.
Comet is deliberately not relocated: its native library binds through JNI
symbols named
`Java_org_apache_comet_Native_*` and resolves shared objects as classpath
resources under
`org/apache/comet/<os>/<arch>/`, so relocating it builds cleanly and fails
on the first native call.
### Impact
New artifacts only, no change to any existing module or code path. Three
properties of the bundle
worth knowing:
- **Requires Java 17.** Comet ships class file version 61 bytecode, so the
bundle cannot load on
Java 11 even for Spark 3.4 and 3.5. Bundle validation runs it only on the
Java 17 pass.
- **Linux only.** Comet ships `linux/aarch64` and `linux/amd64` natives and
nothing else.
- **Large.** About 161 MB per artifact, so roughly 800 MB added per release
across five
coordinates.
Comet leaves the Hudi scan to Spark (`Unsupported file format
HoodieFileGroupReaderBasedFileFormat`), so Parquet decode is not native. With
`spark.comet.convert.parquet.enabled=true` everything above the scan is: a
join of two copy-on-write
tables plans as `CometSortMergeJoin` over
`CometExchange`/`CometNativeShuffle` feeding
`CometHashAggregate`. Measured 0 Comet operators without that config and 10
with it. Copy-on-write
bridges columnar to columnar; merge-on-read reads row by row (`Batched:
false`) and bridges through
`CometSparkRowToColumnar`, which is a per-cell conversion and worth
benchmarking before adopting.
### Risk Level
low
Nothing existing changes behavior; the risk is confined to the build and
release process.
- All seven Spark profiles built. 3.4/3.5/4.0/4.1 produce a bundle; 3.3, 4.2
and a profile-less
build succeed and produce none. `-DdeployArtifacts=true` checked, since it
activates another
profile in the same pom.
- The Spark 3.5 bundle was run in a `linux/arm64` container: both
`libcomet.so` present, Comet
entry points unrelocated, no Comet class relocated under
`org/apache/hudi`, and a Hudi write,
read and join returning expected results with a native plan.
- Bundle validation asserts `CometSortMergeJoin` in the plan of a join over
two Hudi tables, so it
fails if Comet is packaged but not executing, which a row-count check
would not catch.
Only the Spark 3.5 bundle is covered by CI: the `spark4.0` and `spark4.1`
bundle-validation matrix
entries are currently commented out upstream, so those two published bundles
are built but not
exercised.
For the release manager: `libcomet.so` statically links a Rust crate graph
whose licenses Comet's
own jar does not enumerate, so the LICENSE entry here covers Comet and the
protobuf it
redistributes, but not those crates.
### Documentation Update
`packaging/hudi-native-spark-bundle/README.md` covers the support matrix,
the Java 17 and Linux
requirements, the configs that enable Comet, and which parts of a plan are
accelerated. No new
Hudi configs. A website update should follow if this is adopted.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
