I committed a fix to the test - it was just wrong and tested the wrong thing.
Shai On Tue, Sep 30, 2014 at 5:32 PM, Shai Erera <ser...@gmail.com> wrote: > I've tried to change the test to assert better that the facet counts are > sampled OK. E.g. the test currently computes a STDDEV, but that's wrong > since we have only 5 categories that are sampled, and so it's not a real > normal distribution, and every difference from the mean results in a very > high STDDEV... So I tried Pearson's Chi-Squared test ( > http://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test), which works > better on more runs, but still fails from time to time. > > At that point I started thinking what exactly are we testing -- we have a > Random sampler which samples DOCUMENTS not CATEGORIES. So e.g. if I index > documents w/ categories in that order: A B C D A B C D A B C D and so > forth, and I sample every 4th DOCUMENT, I could very well count only > category D, while A, B and C might see none to very low counts, resulting > in big STDDEV, or Chi-Square result. > > The sampler has no knowledge of the indexed categories, only the indexed > documents. And you could easily write an adversary indexer which indexes > categories such that the random sampler almost never samples some > categories. > > Instead I think we should assert that the sampler sampled roughly e.g. 10% > of the docs, irrespective of the counts of the categories. What do you > think? > > On Tue, Sep 23, 2014 at 9:47 PM, Shai Erera <ser...@gmail.com> wrote: > >> This is a test bug, but I'm not yet sure how to fix it. The test verifies >> the sampling works OK by computing some statistics about the counted >> facets. In particular it computes the standard deviation and ensures that >> it's smaller than some arbitrary value (200). However, with this seed and >> test parameters, the standard deviation is 215, and I've verified that with >> any seed, if you fix the number of indexed documents to a high enough >> number (>50,000), it will likely be bigger than 200. >> >> What I'm not sure about is how to fix the test -- increasing the number >> from 200 to 300 will only push the limit further until another failure, >> because of other test parameters. I can do that, and investigate again if >> another run fails. >> >> But increasing that number too high misses the point I think, since if >> our random sampling isn't really random, we'll fail to detect that. >> >> Basically, with some very bad luck, we could sample such that we hit the >> maximum value of the variance, and therefore no matter the value we'll >> compare the standard deviation to, we might run into this extremely >> bad-luck-case at some point. >> >> One choice is to increase the value now, and accept that some runs may >> fail, once in a long while ... >> >> Do we have other tests that do random sampling of stuff and assert the >> sampled values? >> >> Shai >> >> On Mon, Sep 22, 2014 at 10:19 PM, Apache Jenkins Server < >> jenk...@builds.apache.org> wrote: >> >>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/635/ >>> >>> 1 tests failed. >>> REGRESSION: >>> org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling >>> >>> Error Message: >>> >>> >>> Stack Trace: >>> java.lang.AssertionError >>> at >>> __randomizedtesting.SeedInfo.seed([EB7A704156A4175F:162195CE7F3E0E8]:0) >>> at org.junit.Assert.fail(Assert.java:92) >>> at org.junit.Assert.assertTrue(Assert.java:43) >>> at org.junit.Assert.assertTrue(Assert.java:54) >>> at >>> org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:136) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) >>> at >>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >>> at >>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>> at >>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>> at >>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) >>> at >>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >>> at >>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) >>> at >>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>> at >>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >>> at >>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>> at >>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>> at >>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) >>> at >>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>> at >>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >>> at >>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> >>> >>> Build Log: >>> [...truncated 8590 lines...] >>> [junit4] Suite: >>> org.apache.lucene.facet.TestRandomSamplingFacetsCollector >>> [junit4] 2> NOTE: download the large Jenkins line-docs file by >>> running 'ant get-jenkins-line-docs' in the lucene directory. >>> [junit4] 2> NOTE: reproduce with: ant test >>> -Dtestcase=TestRandomSamplingFacetsCollector >>> -Dtests.method=testRandomSampling -Dtests.seed=EB7A704156A4175F >>> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true >>> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt >>> -Dtests.locale=nl_NL -Dtests.timezone=VST -Dtests.file.encoding=ISO-8859-1 >>> [junit4] FAILURE 21.1s J2 | >>> TestRandomSamplingFacetsCollector.testRandomSampling <<< >>> [junit4] > Throwable #1: java.lang.AssertionError >>> [junit4] > at >>> __randomizedtesting.SeedInfo.seed([EB7A704156A4175F:162195CE7F3E0E8]:0) >>> [junit4] > at >>> org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:136) >>> [junit4] > at java.lang.Thread.run(Thread.java:745) >>> [junit4] 2> NOTE: test params are: codec=Lucene410: >>> {$payloads$=FST41, $full_path$=FST41, $facets=FSTOrd41, >>> EvenOdd=PostingsFormat(name=Memory doPackFST= false)}, >>> docValues:{$facets=DocValuesFormat(name=Lucene410)}, >>> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {EvenOdd=DFR >>> I(F)L1}, locale=nl_NL, timezone=VST >>> [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation >>> 1.7.0_65 (64-bit)/cpus=16,threads=1,free=197229168,total=297795584 >>> [junit4] 2> NOTE: All tests run in this JVM: >>> [TestOrdinalMappingAtomicReader, TestCachedOrdinalsReader, >>> TestRandomSamplingFacetsCollector] >>> [junit4] Completed on J2 in 21.20s, 1 test, 1 failure <<< FAILURES! >>> >>> [...truncated 58 lines...] >>> BUILD FAILED >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:498: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:471: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:60: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:462: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:2140: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/module-build.xml:58: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1358: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: >>> There were test failures: 23 suites, 155 tests, 1 failure >>> >>> Total time: 116 minutes 23 seconds >>> Build step 'Invoke Ant' marked build as failure >>> Archiving artifacts >>> Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #633 >>> Archived 5 artifacts >>> Archive block size is 32768 >>> Received 0 blocks and 228817 bytes >>> Compression is 0.0% >>> Took 0.11 sec >>> Recording test results >>> Email was triggered for: Failure >>> Sending email for trigger: Failure >>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >> >> >