> On 15 Oct 2020, at 15:56, John Patrick <nhoj.patr...@gmail.com> wrote: > > <— snip --> > > Will look at vfs and maybe also rng regarding tidying up the tests etc.
Did you mean ’tidying up math’? Regarding RNG, the test suite relies on the fact that statistical tests can fail and be rerun. This uses a feature in JUnit 4 that is applied in the config for the surefire plugin: <rerunFailingTestsCount>2</rerunFailingTestsCount> It applies to the modules: commons-rng-core commons-rng-sampling commons-rng-simple As far as I know there is no equivalent in JUnit 5. The p-values for most tests are set at 1/100. With 2 reruns the fail rate for single tests is 1/100000. If I recall correctly when I looked at this the overall fail rate for the rng-core test suite is around 1/600. I am not sure about the others but there are fewer tests in those and the reruns is set at 1 and not 2. The failures are expected. This is a random sampling library. It cannot be tested with fixed seeds as then it is not fully exercising the randomness or the self-seeding functionality. The result is that any failures on the CI system just have to be looked at across the different Java builds. If the failures are in the same place it should be looked at, otherwise the job can be restarted and should pass second time. Currently it is more helpful to leave the test suite as is rather than change the p-value to 1/100000 for tests which is what would be required on JUnit 5 without reruns. Without change to the test suite the back log of CI results can actually be used to analyse the failure statistics (although I’ve not done this non-locally). So until JUnit 5 offers a reruns facility the easiest action is to remain at JUnit 4. Any thoughts on this with regard to an upgrade path are welcome. Alex
