yihua opened a new pull request, #19082:
URL: https://github.com/apache/hudi/pull/19082
### Describe the issue this Pull Request addresses
The `Java CI` workflow runs the full Spark datasource test suite once per
Spark version and Java/Scala track, producing a large number of checks and high
compute cost. Measured per-job timings from a recent master run show ~2,119
runner-minutes per push, ~90% of it in the per-version Spark datasource jobs,
most of which re-run Spark-version-independent tests on every version.
This PR runs a curated **core** subset on every Spark version and the
**full** suites only on the latest 3.x (spark3.5) and latest 4.x (spark4.2).
### Summary and Changelog
- New `core-tests` Maven profile: surefire `groups=core` plus scalatest
`tagsToInclude=org.apache.hudi.functional.SparkSQLCoreFlow`, injected only in
this profile (an empty `tagsToInclude` would emit `-n ""` and run zero scala
tests).
- The core set is excluded from the normal tasks so nothing runs twice on a
version: `core` added to `excludedGroups` of
`unit-tests`/`functional-tests`/`-b`/`-c`, and the scalatest exclude is driven
by the `hoodie.scalatest.tagsToExclude` property (default `SparkSQLCoreFlow`).
- `CoreFlow` scalatest `Tag` object (name matches the existing
`@SparkSQLCoreFlow` `@TagAnnotation`) for per-test `taggedAs(CoreFlow)`.
- Tagged core set: `@Tag("core")` on representative
`TestCOWDataSource`/`TestMORDataSource` methods (read/write round-trip,
vectorized reader, snapshot/read-optimized/incremental reads, schema evolution)
and the parquet/orc/InternalRow writers; `taggedAs(CoreFlow)` on basic
INSERT/UPDATE/DELETE/MERGE INTO/CREATE TABLE blocks.
- The previously dead-wired `TestSparkSqlCoreFlow` anchor (tagged
`@SparkSQLCoreFlow`, excluded everywhere, included nowhere) is now run by the
core job, trimmed to a representative table-type/metadata/keygen/index spread
so it fits the per-version budget.
- `bot.yml`: full-suite datasource jobs reduced to a single version each;
one matrix-driven job `test-spark-core-tests` (Java 11: spark3.3/3.4/3.5; Java
17: spark3.5/4.0/4.1/4.2, via per-entry `javaVersion`/`mvnProfiles`) builds
once and runs `-Pcore-tests` plus the quickstart on every version. Flink
(already tiered), bundle-validation, docker, and build-only jobs are unchanged.
Estimated impact: ~2,119 -> ~1,142 runner-minutes per push (~46%), and Spark
datasource checks drop from 30 to ~14.
### Impact
CI/test-execution only; no production code or public API change. Non-latest
Spark versions trade the full per-version suite for a curated core subset; full
coverage for all versions can still be restored via a scheduled full-matrix run.
### Risk Level
low
CI/test-tiering change only, no source changes. Opened as draft to validate
the new matrix in CI before review.
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
