voonhous commented on PR #19906: URL: https://github.com/apache/hudi/pull/19906#issuecomment-5638151239
Local three-way A/B on the two classes, single fork, JDK 11, Spark 3.5, same machine, back to back: | setting | TestHoodieClientOnCopyOnWriteStorage | TestMergeIntoTable (47 tests) | |---|---|---| | baseline: shuffle 4, default parallelism 4, AQE on | 267 s | 6m14s | | this PR: 2, 2, AQE off | 270 s | 6m07s | | hybrid: shuffle 2, default parallelism 4, AQE off | 260 s | 9m16s | Two conclusions. The +95 s CI regression on the client test does not reproduce here: this machine has more cores than the 4-vCPU runner, so two tasks per stage do not starve it the way they do there. And the hybrid is not a free improvement: MERGE INTO got slower with it, by more than run-to-run noise on this machine usually is, so it would need its own CI runs to be judged, at about 530 runner-minutes each. Given that, the PR stays at 2/2/AQE off. Net over three CI runs it is -21 runner-minutes against the pool's own drift, with fta as the one job that pays about 6 minutes for it; fta is not on the run's critical path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
