[
https://issues.apache.org/jira/browse/LUCENE-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288418#comment-14288418
]
Hoss Man commented on LUCENE-6194:
----------------------------------
bq. Note, however, that if you provide both parts and run multiple iterations
then all of the iterations will be repeated with exactly the same randomness.
yes, but then it would be doing exactly what you asked it too - it's doing many
iterations with both a fixed master seed and a fixed iteration seed.
{quote}
This behavior is useful to detect non-deterministic tests. Note the difference
– if your test fails some iterations with:
-Dtests.seed=dead -Dtests.iters=100
it means for "certain" seeds (at certain iterations) the code is failing, while
if your test case fails intermittently on:
-Dtests.seed=dead:beef -Dtests.iters=100
then there is something non-related to the randomness that is causing the
failure for a particular seed (dead:beef).
{quote}
...i think the best course of action is to assume the test is deterministic,
and include the full seed since that's the fastest way to "reproduce" a failure
in code that was triggered by the randomly generated "input"
if it doesn't reproduce, then the user (who must already be relatively familiar
with teh test framework if he was even trying test.iters in hte first place)
will have a good indication that hte test is flawed.
---
to me, including the full seed seems like th best course of action for hte
reproduce line, in particular because of how the reproduce lines (plural) are
reported when you use test.iters in the first place.
consider this diff...
{noformat}
Index: core/src/test/org/apache/lucene/TestSearch.java
===================================================================
--- core/src/test/org/apache/lucene/TestSearch.java (revision 1654092)
+++ core/src/test/org/apache/lucene/TestSearch.java (working copy)
@@ -34,6 +34,11 @@
/** JUnit adaptation of an older test case SearchTest. */
public class TestSearch extends LuceneTestCase {
+ public void testReproduceLine() throws Exception { // nocommit
+ assertTrue(0 < random().nextInt(100)); // will fail on '0'
+ }
+
+
public void testNegativeQueryBoost() throws Exception {
Query q = new TermQuery(new Term("foo", "bar"));
q.setBoost(-42f);
{noformat}
...and this output...
{noformat}
hossman@frisbee:~/lucene/dev/lucene/core$ ant test -Dtestcase=TestSearch
-Dtests.method=testReproduceLine\* -Dtests.iters=500
Buildfile: /home/hossman/lucene/dev/lucene/core/build.xml
[junit4] <JUnit4> says שלום! Master seed: 6DAA9E4F7FABD6FC
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(16679@frisbee).
[junit4] Suite: org.apache.lucene.TestSearch
[junit4] OK 0.08s | TestSearch.testReproduceLine {#0
seed=[6DAA9E4F7FABD6FC:F8FFD9389F6A3705]}
[junit4] OK 0.00s | TestSearch.testReproduceLine {#1
seed=[6DAA9E4F7FABD6FC:4CA965C4ABA8FC29]}
[junit4] OK 0.01s | TestSearch.testReproduceLine {#2
seed=[6DAA9E4F7FABD6FC:C240F318FA6CB4E2]}
[junit4] OK 0.01s | TestSearch.testReproduceLine {#3
seed=[6DAA9E4F7FABD6FC:F3AE58FD9692EFCB]}
[junit4] OK 0.00s | TestSearch.testReproduceLine {#4
seed=[6DAA9E4F7FABD6FC:BF6FDD50379A2F70]}
[junit4] OK 0.01s | TestSearch.testReproduceLine {#5
seed=[6DAA9E4F7FABD6FC:2E950E0F4A266270]}
...
[junit4] OK 0.01s | TestSearch.testReproduceLine {#190
seed=[6DAA9E4F7FABD6FC:245CC0506485B2F4]}
[junit4] OK 0.01s | TestSearch.testReproduceLine {#191
seed=[6DAA9E4F7FABD6FC:A72F9FDAAB585D2E]}
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSearch
-Dtests.method=testReproduceLine -Dtests.seed=6DAA9E4F7FABD6FC
-Dtests.slow=true -Dtests.locale=lt_LT -Dtests.timezone=America/Cuiaba
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.14s | TestSearch.testReproduceLine {#192
seed=[6DAA9E4F7FABD6FC:E53D5B74EA5CDF58]} <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at
__randomizedtesting.SeedInfo.seed([6DAA9E4F7FABD6FC:E53D5B74EA5CDF58]:0)
[junit4] > at
org.apache.lucene.TestSearch.testReproduceLine(TestSearch.java:38)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] OK 0.00s | TestSearch.testReproduceLine {#193
seed=[6DAA9E4F7FABD6FC:B1832CB6030655C9]}
...
[junit4] OK 0.00s | TestSearch.testReproduceLine {#225
seed=[6DAA9E4F7FABD6FC:676822DBF04739BE]}
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSearch
-Dtests.method=testReproduceLine -Dtests.seed=6DAA9E4F7FABD6FC
-Dtests.slow=true -Dtests.locale=lt_LT -Dtests.timezone=America/Cuiaba
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.01s | TestSearch.testReproduceLine {#226
seed=[6DAA9E4F7FABD6FC:1C72A218C788A63A]} <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at
__randomizedtesting.SeedInfo.seed([6DAA9E4F7FABD6FC:1C72A218C788A63A]:0)
[junit4] > at
org.apache.lucene.TestSearch.testReproduceLine(TestSearch.java:38)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] OK 0.00s | TestSearch.testReproduceLine {#227
seed=[6DAA9E4F7FABD6FC:E7C00D558B908387]}
...
[junit4] OK 0.00s | TestSearch.testReproduceLine {#311
seed=[6DAA9E4F7FABD6FC:4A8C6A00C2513040]}
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSearch
-Dtests.method=testReproduceLine -Dtests.seed=6DAA9E4F7FABD6FC
-Dtests.slow=true -Dtests.locale=lt_LT -Dtests.timezone=America/Cuiaba
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.02s | TestSearch.testReproduceLine {#312
seed=[6DAA9E4F7FABD6FC:4357B0D76C4DC156]} <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at
__randomizedtesting.SeedInfo.seed([6DAA9E4F7FABD6FC:4357B0D76C4DC156]:0)
[junit4] > at
org.apache.lucene.TestSearch.testReproduceLine(TestSearch.java:38)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] OK 0.00s | TestSearch.testReproduceLine {#313
seed=[6DAA9E4F7FABD6FC:427ED3795B660F]}
...
{noformat}
...what's the point of including the reproduce line for every individual
failure if they are all identicle and don't actually include enough context to
reproduce? Adding -Dtests.iters=N to the reproduce line seems like it would be
a lot less helpful then adding the full seed so that i have several individual
commands to try and reproduce -- and as long as those several commands
reproduce reliably, i can post then in jira/email for diagnosing the problem
with the code ... if they _don't_ reproduce, *then* my original execuction line
with -Dtests.iters=N is going to be very useful for getting to the bottom of
how/why the test is broken
> Reproduce with should include -Dtests.iters=N
> ---------------------------------------------
>
> Key: LUCENE-6194
> URL: https://issues.apache.org/jira/browse/LUCENE-6194
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/test-framework
> Reporter: Michael McCandless
> Priority: Minor
>
> I was playing with the patch on LUCENE-6066, and hit a test failure, but then
> the "Reproduce with: " failed to reproduce, which threw me for a bit of a
> loop, until I realized it was missing the -Dtests.iters=N.
> To see this, apply the patch on LUCENE-6066 to trunk and then run this:
> {noformat}
> ant test -Dtestcase=TestPriorityQueue
> -Dtests.method=testRemovalsAndInsertions* -Dtests.seed=2576D26D551748E5
> -Dtests.slow=true
> -Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt
> -Dtests.locale=ru_RU -Dtests.timezone=Europe/Paris -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8 -Dtests.iters=10
> {noformat}
> (or just insert a random test failure into e.g. TestDemo or something).
> It should fail, but the "Reproduce with: ..." line does not include
> -Dtests.iters=10. I think it should?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
