Dear devs, I just merged FLINK-19520, which adds randomization to all tests that use the MiniCluster(WithResource). The overall goal is to provide a faster and better coverage for future changes in the network stack in particular in conjunction with checkpointing.
We implemented it in a way to fulfill these requirements: - Randomization should be seeded in a way that both builds on Azure pipelines and local builds will result in the same settings to ease debugging and ensure reproducibility. - Randomized options should be shown in the test log. - Execution order of test cases will not influence the randomization. - Randomization is hidden, no change on any test is needed. - Randomization only happens during local/remote test execution. User deployments are not affected. - Test developers are able to avoid randomization by explicitly providing values for the randomized configs. To achieve these goals, we use the git commit id of HEAD + test name to calculate the seed of the randomization. So if you experience a test failure on AZP and checkout the same branch/commit, you will get the exact same configuration. If for some reason, git commit id changed (e.g., you added some bugfix commit on top and want to test it) or git commit id cannot be extracted, you can overwrite the seed with the maven property <test.randomization.seed> (test name will still be used additionally to that value). Lastly, you can also add your own randomizations. [1] But please use it conservatively until we have gained enough experience with the setup. Best, Arvid [1] https://github.com/AHeise/flink/blob/3d69b21f05abca0de64a2e778c292a7d0694b263/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/TestStreamEnvironment.java#L97
