voonhous opened a new pull request, #19892: URL: https://github.com/apache/hudi/pull/19892
### Describe the issue this Pull Request addresses Part of the CI improvement epic #19524 (wall-clock stream, #19528). Stacked on #19890 (flatten the job graph); the first commit here is that PR's commit and drops out on rebase once it merges. The longest Java CI job is 73 to 77 minutes because each job runs one surefire fork on a 4-vCPU runner and the job splits follow module boundaries rather than measured time. The top 100 slowest tests in that job add up to 29 of its 67 test minutes; the rest is thousands of few-second tests run one after another. ### Summary and Changelog Opened with the first two commits; the resharding and the benchmark exclusion below land as further commits on this PR once their local validation finishes. - **Two surefire forks per module** (`hudi.surefire.forkCount`, default 2, override with `-D...=1`). The two fixed listen ports in the test harnesses, HiveServer2 at 9999 and the test Zookeeper at 2828, are now chosen once per JVM, the way the metastore and HDFS ports already were, so two forks in one module do not collide. JaCoCo's agent locks the exec file, so the forks share one. - **TestHoodieDeltaStreamer split into four classes by area.** It was 102 tests and 35 of the utilities job's 51 test minutes in one class, which a second fork cannot help with. Bodies moved verbatim; the three helpers that read base-class state moved to HoodieDeltaStreamerTestBase; the nested transformers stay where the seven external references expect them. - (follows) **Spark workflow resharded by measured time.** Java UT split by package (functional versus the rest, the three fat classes with FTC), FTA and FTB as their own jobs, Scala DML and "other" each split by package, utilities split into streamer and rest. 14 jobs, under the ASF target of 15. - (follows) The hudi-common bloom filter benchmark is tagged `benchmark` and excluded from the unit-tests profile. Its javadoc said it never runs in CI, but the CI jobs' `-Dtest` exclusion patterns widen surefire's includes to every class, and it cost 100 seconds in every job that runs hudi-common tests. ### Impact Measured locally under JDK 11 with two forks: the whole hudi-hive-sync suite passed with no bind failures; the four DeltaStreamer classes ran in 13 minutes wall clock against 24 minutes of summed class time. The first CI run of this PR measures the fork change alone; the resharding commit shows its own delta on top. Each fork gets the full `argLine` heap, so a runner can now commit up to 8 GB where it committed 4 GB. `-Xms` is 128m, so this is demand-driven; if a lane starts OOM-killing, `-Dhudi.surefire.forkCount=1` on that lane is the escape hatch. <details> <summary>Measurements behind the resharding (master run 34314808833, single fork)</summary> | job | test time | dominant chunk | |---|---|---| | test-utilities | 51 min | TestHoodieDeltaStreamer alone 35 min | | java17 part1 (Java UT) | 67 min | functional package 32 min, rest 35 min | | java17 part2 | 53 min | FTA 35 min, three fat classes 18 min | | java17 part3 | 40 min | FTB 29 min, FTC 12 min | | java17 scala-dml | 54 min | dml.others 30 min | | java17 scala-other | 51 min | procedure 16, index 9, ddl 9, functional 6 | Per-class times came from surefire lines in the job logs; the next rebalance should re-measure the same way. </details> ### Risk Level medium Fork isolation is the risk: a test that binds a fixed port or writes a fixed path outside the temp dir would now collide with its neighbour. The audit found the two ports above and nothing else; `-Dhudi.surefire.forkCount=1` reverts the behaviour without a code change. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
