voonhous opened a new pull request, #19906: URL: https://github.com/apache/hudi/pull/19906
### Describe the issue this Pull Request addresses Part of the CI improvement epic #19524 (#19528). Stacked on #19892; the commits before "test: run Spark tests with two shuffle partitions and AQE off" are that PR's and drop out on rebase. Idea taken from #18201. The two shared Spark test providers, `HoodieClientTestUtils.getSparkConfForTest` and `SparkProvider`, run every test with four shuffle partitions and adaptive query execution on. A test writes a few hundred rows, so most of those partitions are empty and the adaptive planner has nothing to optimise; both are scheduling overhead paid on every shuffle of every test. hudi-utilities already runs its tests at two. ### Summary and Changelog - `spark.sql.shuffle.partitions` and `spark.default.parallelism` go from 4 to 2 and `spark.sql.adaptive.enabled` is set to false in both providers. Three lines each, no test changed. ### Impact Measurement PR: the CI run is the number. Local A/B at the same commit, single fork, JDK 11: TestMergeIntoTable 326 s to 248 s, TestInsertTable 171 s to 135 s, the three fat datasource classes within noise. The job-by-job comparison against #19892's run 34499656524 goes here once this PR has run. ### Risk Level low A test that asserts on a file or partition count derived from the shuffle partition count would change its expectation; #18201 had to adjust a couple. The CI run finds them. The three lines revert cleanly. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
