Hi,
What do you think about the following change?
https://github.com/apache/iceberg/pull/13675
Background:
We build modules in parallel, but test execution is serial in Iceberg.
I found that increasing the test parallelism for a module greatly
reduces the test execution time. For example, on my machine (MacBook
Pro M1 Max) it takes ~32 minutes to execute the Flink tests via:
./gradlew :iceberg-flink:iceberg-flink-2.1:check
But if I increase `maxParallelForks` in the test section of the Flink
build file to 8, it will only take 9 minutes, which is a speedup of
3.5x.
Instead of doing this manually, we can do it via the property added in above PR:
./gradlew :iceberg-flink:iceberg-flink-2.1:check -DtestParallelism=8
I also added an 'auto' config where we take half of the available CPUs, e.g.:
./gradlew :iceberg-flink:iceberg-flink-2.1:check -DtestParallelism=auto
It is important to note that any modules which already set
`maxParallelForks` will not be affected by this change. So we can
still enforce non-parallel test execution. Also, the change is opt-in
and doesn't change the default build configuration.
Any objections against merging this functionality?
Cheers,
Max
