Thanks for approving the PR! I just ran a test with Spark 4.0:
Command: ./gradlew clean :iceberg-spark:iceberg-spark-4.0:check Completion time: 46m 7s Command: ./gradlew clean :iceberg-spark:iceberg-spark-4.0:check -DtestParallelism=8 Completion time: 22m 3s Just a speedup of ~2 here, but still considerable. Cheers, Max On Tue, Nov 18, 2025 at 6:10 PM Steven Wu <[email protected]> wrote: > > This seems like a great improvement to cut build time. The results are very > encouraging. We will have to see if the parallel tests increase the test > flakiness. > > Can we also test it with Spark? Spark and Flink are the heavy modules. But we > can roll out the changes in phases. > > > > On Tue, Nov 18, 2025 at 7:53 AM Maximilian Michels <[email protected]> wrote: >> >> Hi Eduard, >> >> That's a good point. I just ran the following command which completed >> in 3 min 27 sec: >> >> ./gradlew clean :iceberg-core:check -DtestParallelism=8 >> >> Omitting the testParallelism flag, the same build target takes 13 min 29 sec. >> >> We see a similar speedup as for the Flink tests. It does look like we >> would benefit from this feature in other modules. In the PR, I enabled >> the parallel CI test execution for Flink, we could also enable it for >> iceberg-core, to allow for faster feedback on the PRs. Of course, the >> general caveats with parallel test execution apply. We will have to >> check which other modules can safely run tests in parallel without any >> flakiness. >> >> Cheers, >> Max >> >> On Tue, Nov 18, 2025 at 11:59 AM Eduard Tudenhöfner >> <[email protected]> wrote: >> > >> > Thanks Max for looking into this. Have we tried increasing the parallelism >> > for other modules than Flink and did those pass successfully? Given that >> > we're running more and more tests with more format versions, it would be >> > quite helpful to bring CI time down if possible. >> > >> > >> > On Tue, Nov 18, 2025 at 12:29 AM Rodrigo Meneses <[email protected]> >> > wrote: >> >> >> >> +1 (non-binding) >> >> >> >> as reducing test execution time will improve developer productivity! >> >> >> >> >> >> On Mon, Nov 17, 2025 at 5:12 AM Maximilian Michels <[email protected]> >> >> wrote: >> >>> >> >>> Hi, >> >>> >> >>> What do you think about the following change? >> >>> https://github.com/apache/iceberg/pull/13675 >> >>> >> >>> Background: >> >>> >> >>> We build modules in parallel, but test execution is serial in Iceberg. >> >>> I found that increasing the test parallelism for a module greatly >> >>> reduces the test execution time. For example, on my machine (MacBook >> >>> Pro M1 Max) it takes ~32 minutes to execute the Flink tests via: >> >>> >> >>> ./gradlew :iceberg-flink:iceberg-flink-2.1:check >> >>> >> >>> But if I increase `maxParallelForks` in the test section of the Flink >> >>> build file to 8, it will only take 9 minutes, which is a speedup of >> >>> 3.5x. >> >>> >> >>> Instead of doing this manually, we can do it via the property added in >> >>> above PR: >> >>> >> >>> ./gradlew :iceberg-flink:iceberg-flink-2.1:check -DtestParallelism=8 >> >>> >> >>> I also added an 'auto' config where we take half of the available CPUs, >> >>> e.g.: >> >>> >> >>> ./gradlew :iceberg-flink:iceberg-flink-2.1:check >> >>> -DtestParallelism=auto >> >>> >> >>> It is important to note that any modules which already set >> >>> `maxParallelForks` will not be affected by this change. So we can >> >>> still enforce non-parallel test execution. Also, the change is opt-in >> >>> and doesn't change the default build configuration. >> >>> >> >>> Any objections against merging this functionality? >> >>> >> >>> Cheers, >> >>> Max
