voonhous opened a new pull request, #19919:
URL: https://github.com/apache/hudi/pull/19919

   ### Describe the issue this Pull Request addresses
   
   Part of the CI improvement epic #19524 (#19527), tracked on #19910. Stacked 
on #19918; the commits before "ci: build and verify only the IT modules in 
integration-tests" belong to that stack and drop out on rebase.
   
   The integration-tests job builds all 66 modules of the reactor (8.9 min), 
then runs `mvn verify` over the same 66 modules, which packages every one of 
them a second time, the 17 bundles included, before running the integration 
tests (21 min). The integration tests live in five modules.
   
   ### Summary and Changelog
   
   - `IT_MODULES` names the five modules with integration-test classes outside 
hudi-flink, whose ITs run in the flink jobs: hudi-cli, hudi-integ-test, 
hudi-aws, hudi-azure, and hudi-gcp-bundle (its `ITGcpBundleCompatibility` came 
with #20020). The docker demo suites in hudi-integ-test are skipped and 
`@Disabled`, and the job has no docker-compose activity.
   - The build step passes them with `-am`. Maven resolves that to 31 modules: 
the core chain through hudi-utilities, the utilities bundle for hudi-cli, the 
hadoop-mr, hive-sync and spark bundles plus two docker pom modules for 
hudi-integ-test, and hudi-gcp-bundle itself (hudi-gcp is already in the chain).
   - The IT step verifies just the five, instead of the whole reactor minus 
hudi-flink.
   
   Every module no longer built here is built, and where a bundle shaded and 
run, in the job that owns it:
   
   | no longer built in integration-tests | already built and tested or 
validated in |
   |---|---|
   | hudi-flink, hudi-flink-client, the Flink datasource modules | test-flink-1 
and test-flink-2 |
   | hudi-flink-bundle and the other ten bundles the ITs do not need | 
validate-bundles |
   | hudi-examples-common, java, k8s | validate-bundles |
   | hudi-examples-spark | ut-other and validate-bundles |
   | hudi-examples-flink | test-flink-1 and validate-bundles |
   | hudi-kafka-connect | test-common-and-other-modules (tests), 
validate-bundles (bundle) |
   | hudi-datahub-sync, hudi-adb-sync | test-common-and-other-modules |
   | metaserver modules | test-common-and-other-modules, validate-bundles |
   | the verify step's second packaging of all 66 modules | this job's own 
build step |
   
   What stays: the 31-module build and all 15 integration-test classes the job 
runs today (8 in hudi-cli, 2 in hudi-aws, 1 in hudi-azure, 1 in 
hudi-gcp-bundle, hudi-integ-test's skipped ones).
   
   ### Impact
   
   Expected against #19892's run 
([34499656559](https://github.com/apache/hudi/actions/runs/34499656559)): build 
step 8.9 to about 5 minutes, verify step 21 to about 16, the job from 33-38 
minutes to the high twenties, about 10 runner-minutes per run. The job sits on 
the plateau with the Scala shards, so the wall clock follows only if those come 
down too. Measured on 
[34670498928](https://github.com/apache/hudi/actions/runs/34670498928), four 
attempts, against #19918's run 
[34668461644](https://github.com/apache/hudi/actions/runs/34668461644), which 
has the same code minus this change:
   
   | integration-tests | #19918 run | this PR (attempts 1 to 4) |
   |---|---|---|
   | Build Project step (s) | 646 | 479, 456, 515, 495 |
   | IT step (s) | 1390 | 1028, 971, 985, 1006 |
   | modules built / verified | 66 / 64 | 30 / 4 |
   | job execution (min) | 37 | 28, 25, 27, 28 |
   
   The IT step runs the same 25 test classes in both logs (the 14 
integration-test classes plus hudi-integ-test's 11 unit classes), with the same 
per-module totals. The run's 19 test jobs total 539, 531, 551 and 516 
runner-minutes over the four attempts (mean 534), against 541 for #19918's run. 
The other 18 jobs each drift by up to 10 minutes between attempts with no code 
change (attempt 3: fta and scala-dml-2 both +10; attempt 4: utilities-streamer 
-10), so a single total cannot resolve a 10-minute change in one job; the job's 
own steps are the measurement. Across about 90 earlier samples of this job on 
master and PRs since 2026-09-04, the IT step never ran under 1258 s. Longest 
job unchanged at 40 to 41 min. Per-job tables in the comments below. These runs 
predate the rebase onto #20020, which adds hudi-gcp-bundle to both steps (one 
more module built and verified, one more IT class).
   
   ### Risk Level
   
   low
   
   Both `-pl` lists validate against the job's profiles, and `-am` makes the 
build's reactor exactly the dependency closure of what verify runs, so nothing 
verify needs can be missing. If a module gains integration tests, `IT_MODULES` 
is the one line to extend (hudi-gcp-bundle was added here when #20020 gave it 
an integration test); the failsafe class count to compare after this PR is 15.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to