I agree not being able to quickly reproduce a failure is bad, but, worse, I don't want to lose test coverage.
The big line doc file has much more content (~200X larger) than the checked in tiny one. Some tests, especially when running nightly, can fully consume the small line docs file and then they'll "wrap around" and won't see unique terms anymore. Does the failure reproduce if you use the big line docs file? (-Dtest.linedocsfile=...)? Ie, it's still reproducible, but it's a hassle because you need the big line file docs locally? If so... then maybe all that's missing is the "repro" line adding the -Dtest.linedocsfile=/path/to/big/line/file/docs? This way, when we try to repro locally, we'd get a hard but clear error (because the path presumably doesn't work in our local env), and we'd know to edit that to point to our local copy? Separately: does the test fail if you beast it using only the tiny line docs file? This really boils down to a test coverage vs test reproducibility tradeoff... but I would favor coverage here: I'd rather know about a bug, but struggle to reproduce/fix it, than not know about it at all. Ignorance isn't bliss... Mike McCandless http://blog.mikemccandless.com On Fri, Mar 23, 2012 at 11:36 PM, Robert Muir <rcm...@gmail.com> wrote: > This is likely a real bug (from added tests in LUCENE-3905) > > But, I cannot reproduce locally since I dont have the special linedocs file. > > I think we need to address this: we cant use random tests if they > arent reproducible... hell i'd even bet happier if linedocsfile took a > URL and the "reproduce-with" actually downloaded the huge file? But I > think we can do better without this... > > For now maybe we should disable the special linedocs file in hudson? > By using it, it means none of our analysis tests are reproducible > anymore... which is a big loss. > > Index: hudson-lucene-trunk.sh > =================================================================== > --- hudson-lucene-trunk.sh (revision 1304701) > +++ hudson-lucene-trunk.sh (working copy) > @@ -33,6 +33,7 @@ > #mv -f dist/*-src.tgz $ARTIFACTS > > # run tests > +# TODO: make reproducible -Dtests.linedocsfile=$TEST_LINE_DOCS_FILE \ > cd $WORKSPACE/$CORE_DIR > $ANT_HOME/bin/ant \ > -Dsvnversion.exe=$SVNVERSION_EXE \ > @@ -40,12 +41,12 @@ > -Dversion=$VERSION \ > -Dtests.multiplier=$TESTS_MULTIPLIER \ > -Dtests.nightly=true \ > - -Dtests.linedocsfile=$TEST_LINE_DOCS_FILE \ > -Dargs="$TEST_JVM_ARGS" \ > test > # release it > mv -f dist/*.tgz $ARTIFACTS > > +# TODO: make reproducible -Dtests.linedocsfile=$TEST_LINE_DOCS_FILE \ > cd $WORKSPACE/$MODULES_DIR > $ANT_HOME/bin/ant \ > -Dsvnversion.exe=$SVNVERSION_EXE \ > @@ -53,7 +54,6 @@ > -Dversion=$VERSION \ > -Dtests.multiplier=$TESTS_MULTIPLIER \ > -Dtests.nightly=true \ > - -Dtests.linedocsfile=$TEST_LINE_DOCS_FILE \ > -Dargs="$TEST_JVM_ARGS" \ > test > > Index: hudson-lucene-3.x.sh > =================================================================== > --- hudson-lucene-3.x.sh (revision 1304701) > +++ hudson-lucene-3.x.sh (working copy) > @@ -24,13 +24,13 @@ > mv -f build/docs/api/* $JAVADOCS_ARTIFACTS > > # run tests > +# TODO: make reproducible -Dtests.linedocsfile=$TEST_LINE_DOCS_FILE \ > JAVA_HOME=$JAVA_HOME_16 $ANT_HOME/bin/ant \ > -Dsvnversion.exe=$SVNVERSION_EXE \ > -Dsvn.exe=$SVN_EXE \ > -Dversion=$VERSION \ > -Dtests.multiplier=$TESTS_MULTIPLIER \ > -Dtests.nightly=true \ > - -Dtests.linedocsfile=$TEST_LINE_DOCS_FILE \ > -Dargs="$TEST_JVM_ARGS" \ > test > # release it > > > On Fri, Mar 23, 2012 at 11:16 PM, Apache Jenkins Server > <jenk...@builds.apache.org> wrote: >> Build: https://builds.apache.org/job/Lucene-3.x/680/ >> >> 1 tests failed. >> REGRESSION: >> org.apache.lucene.analysis.charfilter.HTMLStripCharFilterTest.testRandomHugeStrings >> >> Error Message: >> Uncaught exception by thread: Thread[Thread-10,5,] >> >> Stack Trace: >> org.apache.lucene.util.UncaughtExceptionsRule$UncaughtExceptionsInBackgroundThread: >> Uncaught exception by thread: Thread[Thread-10,5,] >> at >> org.apache.lucene.util.UncaughtExceptionsRule$1.evaluate(UncaughtExceptionsRule.java:83) >> at >> org.apache.lucene.util.LuceneTestCase$RememberThreadRule$1.evaluate(LuceneTestCase.java:516) >> at org.junit.rules.RunRules.evaluate(RunRules.java:18) >> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) >> at >> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) >> at >> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:146) >> at >> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50) >> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) >> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) >> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) >> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) >> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) >> at >> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) >> at >> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) >> at >> org.apache.lucene.util.UncaughtExceptionsRule$1.evaluate(UncaughtExceptionsRule.java:74) >> at >> org.apache.lucene.util.StoreClassNameRule$1.evaluate(StoreClassNameRule.java:36) >> at >> org.apache.lucene.util.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:67) >> at org.junit.rules.RunRules.evaluate(RunRules.java:18) >> at org.junit.runners.ParentRunner.run(ParentRunner.java:300) >> at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) >> at >> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) >> at >> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) >> at >> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743) >> Caused by: java.lang.AssertionError: finalOffset expected:<992> but >> was:<991> >> at org.junit.Assert.fail(Assert.java:93) >> at org.junit.Assert.failNotEquals(Assert.java:647) >> at org.junit.Assert.assertEquals(Assert.java:128) >> at org.junit.Assert.assertEquals(Assert.java:472) >> at >> org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:182) >> at >> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:563) >> at >> org.apache.lucene.analysis.BaseTokenStreamTestCase.access$000(BaseTokenStreamTestCase.java:50) >> at >> org.apache.lucene.analysis.BaseTokenStreamTestCase$AnalysisThread.run(BaseTokenStreamTestCase.java:335) >> >> >> >> >> Build Log (for compile errors): >> [...truncated 18091 lines...] >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org > > > > -- > lucidimagination.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
