voonhous commented on issue #19910: URL: https://github.com/apache/hudi/issues/19910#issuecomment-5643134415
Next lever, same shape as #19909: the integration-tests job builds and verifies far more than its integration tests use. What the job does today (run [34602459140](https://github.com/apache/hudi/actions/runs/34602459140), job step times): `mvn clean install` over the whole reactor, 66 modules including all 17 bundles (8.9 min), then `mvn verify -Pintegration-tests -pl !hudi-flink` over the same reactor (21 min). The verify step packages all 66 modules a second time, shading the 17 bundles again (34 shade executions in the job), then runs the integration tests that exist. Those live in four modules: hudi-cli (8 IT classes, 814 s), hudi-aws (2), hudi-azure (1), hudi-integ-test (its two demo suites are skipped and `@Disabled`; the job has no docker-compose activity). What the four modules need, from Maven's own resolution (`-pl hudi-cli,hudi-integ-test,hudi-aws,hudi-azure -am`): 30 modules, the core chain through hudi-utilities plus the utilities bundle (hudi-cli) and the hadoop-mr, hive-sync and spark bundles with two docker pom modules (hudi-integ-test). The 36 modules it would stop building are each built, and where a bundle shaded and run, in the job that owns them: | removed from integration-tests | already built and tested or validated in | |---|---| | hudi-flink, hudi-flink-client, the Flink datasource modules | test-flink-1 and test-flink-2 (build with -am, run their tests) | | hudi-flink-bundle and the other eleven bundles the ITs do not need | validate-bundles (shades every bundle, then runs them) | | hudi-examples-common, java, k8s | validate-bundles | | hudi-examples-spark | ut-other (with its quickstart test), validate-bundles | | hudi-examples-flink | test-flink-1 (with its unit tests), validate-bundles | | hudi-kafka-connect | test-common-and-other-modules (unit tests); validate-bundles (its bundle) | | hudi-datahub-sync, hudi-adb-sync | test-common-and-other-modules (unit tests) | | metaserver modules | test-common-and-other-modules (platform-service build); validate-bundles (metaserver bundle) | | the verify step's second packaging of all 66 modules | the same job's own build step | What stays: the 30-module build and all 14 integration-test classes the job runs today. Expected: the build step from 8.9 to about 5 minutes and the verify step from 21 to about 16, so the job from 33-38 to the high twenties, about 10 runner-minutes per run. It sits on the plateau with the Scala shards, so the wall clock only moves with it if those come down too. Verification before pushing, as for #19909: each `-pl` list validated against its command's reactor, and a pom scan showing nothing kept depends on anything no longer built. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
