> Now I just need to understand why the test failure is no longer
> reproducing lol.
>
This is indeed the hard part!
> Also it's mildly confusing that when you specify tests.iters it prints a
> single test seed if it is actually going to use many different ones?
>
It prints a single seed because it starts from that seed (the static
initialization, that is). But each test has its own starting point derived
from the main seed and the test name (if I recall right). So when you pass
tests.iters=100 and run a single test, any random call in that test
(excluding static hooks and static blocks) should be different on each
iteration. You can try it by adding:
assumeTrue("", RandomizedTest.randomBoolean());
For example (I added it to TestSearch.java):
./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
...
:lucene:core:test (SUCCESS): 100 test(s), 52 skipped
if you modify this to assertTrue, you'll get to see the hierarchical seed
chain for each test that failed - note the first part is constant, the
second is derived for each test iteration:
./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
-Ptests.seed=deadbeef
and two example failures:
- org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:3EDB7869EFFD5034]} (:lucene:core)
Test output:
/Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:3EDB7869EFFD5034]}" -Ptests.jvms=4
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC
-XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100
-Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512
- org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:F44F3D10E8B98D27]} (:lucene:core)
Test output:
/Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:F44F3D10E8B98D27]}" -Ptests.jvms=4
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC
-XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100
-Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512
If you'd like to repeat tests with *the same* starting seed for each test,
you need to pass the full chain, including the second part of the seed. For
example, this will fail 100 times (and not approximately 50% of the times):
./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
-Ptests.seed=deadbeef:F44F3D10E8B98D27
It may seem a bit complicated but it really isn't... I hope! And for 99%
of tests, you'd probably rerun with the first part of the seed and it'd be
sufficient to locate the problem.
The 'beast' task is a bit different because it physically re-launches the
test infrastructure so if you don't fix the initial seed, each started JVM
will have a different "starting" seed for static initializers and hooks.
This may matter for locale randomization, jvm issues or static initializers
that rely on randomness. But most isolated test methods should only rely on
their starting seed (not the "global starting seed").
Dawid
