yihua opened a new pull request, #20434: URL: https://github.com/apache/hudi/pull/20434
### Describe the issue this Pull Request addresses part of #19673 issue: #20176 **Do not merge.** This is a CI experiment stacked on #20170: it runs Hudi's Spark 3.5 / Scala 2.12 test suites with Apache DataFusion Comet enabled as the Spark plugin and the native Hudi scan through hudi-rs turned on, to find out which tests break when Hudi runs on Comet plus hudi-rs. Nothing here is meant to land; the findings feed #20170 and follow-ups. ### Summary and Changelog - Root pom: an empty `comet.test.jvm.args` property appended to the three `argLine` definitions, the Comet `-Dspark.*` flags defined once, and `rerunFailingTestsCount` / `reuseForks` turned into properties so the lane can switch retries off and fork per class. A `comet-hudi-tests` profile excludes the shaded `hadoop-client-api` that orc-shims adds on JDK 17, which otherwise breaks SparkContext creation in hudi-spark-client, hudi-cli and hudi-utilities. - Each Spark test module gains a `comet-hudi-tests` profile (`-Dcomet.hudi.tests`) that sets the JVM flags and adds the Comet jar as a test dependency, plus `hudi-spark-comet` where that does not create a reactor cycle. `arrow-c-data` is declared right behind the Comet jar on purpose: Comet relocates Arrow but ships `org.apache.arrow.c` unrelocated, and the copy Lance brings in otherwise wins and fails Comet's shuffle with a `NoSuchMethodError`. - New workflow `comet_spark_tests.yml`: a `build-comet` job builds the overlaid Comet on Linux from the pin (Rust, JDK 17, portable x86-64-v3 target, cached) and hands the installed artifact to seven test jobs that mirror the commented-out Spark 3.5 jobs from `bot.yml` on JDK 17, with retries off and one fork per test class. Tables are written at table version 8 through a `hudi-defaults.conf` in each job, because the native scan declines table version 10, which is what every test writes by default. Local smoke runs on macOS (JDK 17): three Scala suites in hudi-spark pass with Comet exec and shuffle active and the native Hudi scan converting ten scans; two Java tests in hudi-spark-client run with Comet plugins active. One known crash: Comet's own Parquet scan reading a MiniDFSCluster path goes through its libhdfs bridge and segfaults the forked JVM (`TestMultiFS`), hence per-class forks. ### Impact None if not merged. The profiles and the workflow are inactive without `-Dcomet.hudi.tests`; the `argLine` change appends an empty property. ### Risk Level low ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
