Thanks Max for looking into this. Have we tried increasing the parallelism for other modules than Flink and did those pass successfully? Given that we're running more and more tests with more format versions, it would be quite helpful to bring CI time down if possible.
On Tue, Nov 18, 2025 at 12:29 AM Rodrigo Meneses <[email protected]> wrote: > +1 (non-binding) > > as reducing test execution time will improve developer productivity! > > > On Mon, Nov 17, 2025 at 5:12 AM Maximilian Michels <[email protected]> wrote: > >> Hi, >> >> What do you think about the following change? >> https://github.com/apache/iceberg/pull/13675 >> >> Background: >> >> We build modules in parallel, but test execution is serial in Iceberg. >> I found that increasing the test parallelism for a module greatly >> reduces the test execution time. For example, on my machine (MacBook >> Pro M1 Max) it takes ~32 minutes to execute the Flink tests via: >> >> ./gradlew :iceberg-flink:iceberg-flink-2.1:check >> >> But if I increase `maxParallelForks` in the test section of the Flink >> build file to 8, it will only take 9 minutes, which is a speedup of >> 3.5x. >> >> Instead of doing this manually, we can do it via the property added in >> above PR: >> >> ./gradlew :iceberg-flink:iceberg-flink-2.1:check -DtestParallelism=8 >> >> I also added an 'auto' config where we take half of the available CPUs, >> e.g.: >> >> ./gradlew :iceberg-flink:iceberg-flink-2.1:check >> -DtestParallelism=auto >> >> It is important to note that any modules which already set >> `maxParallelForks` will not be affected by this change. So we can >> still enforce non-parallel test execution. Also, the change is opt-in >> and doesn't change the default build configuration. >> >> Any objections against merging this functionality? >> >> Cheers, >> Max >> >
