Thanks for the explanation. It makes sense that we start with a given
seed and then each iteration is different because it re-uses the same
Random instance (or whatever static state?) without re-initialization?
On Wed, Apr 3, 2024 at 6:09 PM Dawid Weiss <dawid.we...@gmail.com> wrote:
>
>
>> Now I just need to understand why the test failure is no longer reproducing
>> lol.
>
>
> This is indeed the hard part!
>
>>
>> Also it's mildly confusing that when you specify tests.iters it prints a
>> single test seed if it is actually going to use many different ones?
>
>
> It prints a single seed because it starts from that seed (the static
> initialization, that is). But each test has its own starting point derived
> from the main seed and the test name (if I recall right). So when you pass
> tests.iters=100 and run a single test, any random call in that test
> (excluding static hooks and static blocks) should be different on each
> iteration. You can try it by adding:
>
> assumeTrue("", RandomizedTest.randomBoolean());
>
> For example (I added it to TestSearch.java):
>
> ./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
> ...
> :lucene:core:test (SUCCESS): 100 test(s), 52 skipped
>
> if you modify this to assertTrue, you'll get to see the hierarchical seed
> chain for each test that failed - note the first part is constant, the second
> is derived for each test iteration:
>
> ./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
> -Ptests.seed=deadbeef
>
> and two example failures:
>
> - org.apache.lucene.TestSearch.testSearch
> {seed=[DEADBEEF:3EDB7869EFFD5034]} (:lucene:core)
> Test output:
> /Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
> Reproduce with: gradlew :lucene:core:test --tests
> "org.apache.lucene.TestSearch.testSearch {seed=[DEADBEEF:3EDB7869EFFD5034]}"
> -Ptests.jvms=4 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC
> -XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100
> -Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512
>
> - org.apache.lucene.TestSearch.testSearch
> {seed=[DEADBEEF:F44F3D10E8B98D27]} (:lucene:core)
> Test output:
> /Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
> Reproduce with: gradlew :lucene:core:test --tests
> "org.apache.lucene.TestSearch.testSearch {seed=[DEADBEEF:F44F3D10E8B98D27]}"
> -Ptests.jvms=4 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC
> -XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100
> -Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512
>
> If you'd like to repeat tests with *the same* starting seed for each test,
> you need to pass the full chain, including the second part of the seed. For
> example, this will fail 100 times (and not approximately 50% of the times):
>
> ./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
> -Ptests.seed=deadbeef:F44F3D10E8B98D27
>
> It may seem a bit complicated but it really isn't... I hope! And for 99% of
> tests, you'd probably rerun with the first part of the seed and it'd be
> sufficient to locate the problem.
>
> The 'beast' task is a bit different because it physically re-launches the
> test infrastructure so if you don't fix the initial seed, each started JVM
> will have a different "starting" seed for static initializers and hooks. This
> may matter for locale randomization, jvm issues or static initializers that
> rely on randomness. But most isolated test methods should only rely on their
> starting seed (not the "global starting seed").
>
> Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
