[ https://issues.apache.org/jira/browse/LUCENE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated LUCENE-7132: ----------------------------- Attachment: LUCENE-7132.patch W/o really having firm idea what's going on here i spent some time reviewing the tests that Ahmet posted, and then started poking at things with a stick -- mainly to see if I could figure out the missing piece of the puzzle that caused things to fail in "testRajeshData()" but not in any of our existing randomized tests. I feel down the rabit hole a bit looking into this, and I still have no concrete idea what the underlying problem is, but i have a few uneducated guesses... * the problem seems to only affect BooleanQueries where a "coord" factor is involved ** adding {{setDisableCoord(true)}} on the queries in testRajeshData seems to make all seeds pass ** so obviously reproducing this bug requires a Similarity that has a non-trivila coord function (like ClassicSimilaity in Ahmet's TestExplain) * the _score_ values are inconsistent between randomized runs of testRajeshData, while the _explain_ values stay the same ** even though the test data & similarity are fixed, testRajeshData pass fine with some seeds, while others fail ** when testRajeshData does fail, it's because the scores have changed compared to the values produced in previous passing runs ** {color:red}best guess: something about the MergePolicy and the way docs are co-located in diff segments is triggering a pathological code path in some BooleanWeight/Scorer optimization code?{color} *** perhaps because a segment may not even contain one of the optional terms, so it's Scorer is null? ** if it's not the MergePolicy, then perhaps something about the codec used and the term stats? Things i've added/revised in this new version of the patch... * added some more asserts to TestExplain.testRajeshData to help demonstrate that it's the score that has changed between successful vs failing runs * fixed a bug in TestExplain.testExplainScoreEquality that was causing false failures ** {{randomSimpleString(random())}} can produce the empty string, which was causing the test to not match the numDocs it was expecting since a whitespace based analyzer is being used ** NOTE: these false failures where the only case i've ever seen TestExplain.testExplainScoreEquality fail * added a TestBaseExplanationTestCase to the test-framework ** I had some initial concerns that maybe some old changes/refactorings to BaseExplanationTestCase had completely eliminated the checks that were suppose to be done in that test, so i added this class to ensure it would fail as expected when an Explanation didn't match scores * added more coord randomization in TestSimpleExplanations ** even w/these changes, i've never seen these tests fail * added a new TestSimpleExplanationsWithHeavyIDF ** this was one of the first things i tried, based on Ahmet's concerns about {{IDFStats.normalize}} ** since TestExplain.testRajeshData was failing so consistently, but we've never seen failures like this from any of of the existing BaseExplanationTestCase, i speculated that maybe IDF stats or index size were a big factor, and tried to bang out a subclass of TestSimpleExplanations that added a lot of (ignored) docs with the same terms to make the doc freq stats more interesting for the terms being searched on ** i've never seen this test fail ** some refactoring in BaseExplanationTestCase was needed to write this test ---- As things stand now with the current patch, here are some test seeds for TestExplain that pass for me... {noformat} ant test -Dtestcase=TestExplain -Dtests.method=testRajeshData -Dtests.seed=11679FC4F397A674 -Dtests.slow=true -Dtests.locale=th -Dtests.timezone=Etc/GMT+1 -Dtests.asserts=true -Dtests.file.encoding=UTF-8 -Dtests.verbose=true ... [junit4] 2> NOTE: test params are: codec=Asserting(Lucene60): {id=PostingsFormat(name=SimpleText), body=PostingsFormat(name=Memory doPackFST= false)}, docValues:{}, maxPointsInLeafNode=604, maxMBSortInHeap=6.961830329642295, sim=ClassicSimilarity, locale=th, timezone=Etc/GMT+1 {noformat} {noformat} ant test -Dtestcase=TestExplain -Dtests.verbose=true -Dtests.seed=610EF558241D1898 -Dtests.slow=true -Dtests.locale=th -Dtests.timezone=Etc/GMT+1 -Dtests.asserts=true -Dtests.file.encoding=UTF-8 -Dtests.verbose=true ... [junit4] 2> NOTE: test params are: codec=Asserting(Lucene60): {id=PostingsFormat(name=Memory doPackFST= true), body=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=738, maxMBSortInHeap=5.910307356558968, sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=th, timezone=Etc/GMT+1 {noformat} And here are some examples of failures... fails... {noformat} ant test -Dtestcase=TestExplain -Dtests.method=testRajeshData -Dtests.seed=A2E2DCFC7459241E -Dtests.slow=true -Dtests.locale=ar-EG -Dtests.timezone=Africa/Asmara -Dtests.asserts=true -Dtests.file.encoding=US-ASCII -Dtests.verbose=true [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestExplain -Dtests.method=testRajeshData -Dtests.seed=A2E2DCFC7459241E -Dtests.slow=true -Dtests.locale=ar-EG -Dtests.timezone=Africa/Asmara -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] FAILURE 1.09s | TestExplain.testRajeshData <<< [junit4] > Throwable #1: java.lang.AssertionError: o-365 score expected:<0.7475659251213074> but was:<0.2491886466741562> [junit4] > at __randomizedtesting.SeedInfo.seed([A2E2DCFC7459241E:C3C4FAF539C031CF]:0) [junit4] > at org.apache.lucene.search.TestExplain.testRajeshData(TestExplain.java:210) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=ar-EG, timezone=Africa/Asmara [junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=236148720,total=248512512 {noformat} {noformat} ant test -Dtestcase=TestExplain -Dtests.method=testRajeshData -Dtests.seed=92DC100B28B70C2C -Dtests.slow=true -Dtests.locale=he -Dtests.timezone=America/Guyana -Dtests.asserts=true -Dtests.file.encoding=UTF-8 -Dtests.verbose=true [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestExplain -Dtests.method=testRajeshData -Dtests.seed=92DC100B28B70C2C -Dtests.slow=true -Dtests.locale=he -Dtests.timezone=America/Guyana -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] FAILURE 1.03s | TestExplain.testRajeshData <<< [junit4] > Throwable #1: java.lang.AssertionError: m-o-365 score expected:<1.2357021570205688> but was:<0.41190072894096375> [junit4] > at __randomizedtesting.SeedInfo.seed([92DC100B28B70C2C:F3FA3602652E19FD]:0) [junit4] > at org.apache.lucene.search.TestExplain.testRajeshData(TestExplain.java:207) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] 2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, locale=he, timezone=America/Guyana [junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=173898752,total=249561088 [junit4] 2> NOTE: All tests run in this JVM: [TestExplain] {noformat} My refactoring of BaseExplanationTestCase also seems to have somehow introduced/tickled a stranage bug in the test framework that trips our SecurityManager settings -- note that the seed below causes an odd AccessControlException at the suite level *after* all TestSimpleExplanations test methods have finished successfully... {noformat} ant test -Dtestcase=TestSimpleExplanations -Dtests.seed=74B719CE50C8168A -Dtests.slow=true -Dtests.locale=sr-Latn -Dtests.timezone=America/St_Vincent -Dtests.asserts=true -Dtests.file.encoding=UTF-8 ... [junit4] OK 0.01s | TestSimpleExplanations.testBQ14 [junit4] OK 0.01s | TestSimpleExplanations.testBQ1 [junit4] OK 0.03s | TestSimpleExplanations.testBQ20 [junit4] OK 0.03s | TestSimpleExplanations.testBQ4 [junit4] OK 0.01s | TestSimpleExplanations.testP2 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene60), sim=RandomSimilarity(queryNorm=false,coord=no): {field=DFR I(n)B2, alt=DFR I(ne)B1}, locale=sr-Latn, timezone=America/St_Vincent [junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=303553832,total=329777152 [junit4] 2> NOTE: All tests run in this JVM: [TestSimpleExplanations] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSimpleExplanations -Dtests.seed=74B719CE50C8168A -Dtests.slow=true -Dtests.locale=sr-Latn -Dtests.timezone=America/St_Vincent -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 0.00s | TestSimpleExplanations (suite) <<< [junit4] > Throwable #1: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.nio.fs") [junit4] > at __randomizedtesting.SeedInfo.seed([74B719CE50C8168A]:0) [junit4] > at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) [junit4] > at java.security.AccessController.checkPermission(AccessController.java:884) [junit4] > at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) [junit4] > at java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1564) [junit4] > at java.lang.Class.checkPackageAccess(Class.java:2372) [junit4] > at java.lang.Class.checkMemberAccess(Class.java:2351) [junit4] > at java.lang.Class.getDeclaredFields(Class.java:1915) [junit4] > at java.security.AccessController.doPrivileged(Native Method) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] Completed [1/1 (1!)] in 2.50s, 68 tests, 1 error <<< FAILURES! {noformat} > ScoreDoc.score() returns a different value than that of Explanation's > --------------------------------------------------------------------- > > Key: LUCENE-7132 > URL: https://issues.apache.org/jira/browse/LUCENE-7132 > Project: Lucene - Core > Issue Type: Bug > Components: core/search > Affects Versions: 5.5 > Reporter: Ahmet Arslan > Assignee: Steve Rowe > Attachments: LUCENE-7132.patch, LUCENE-7132.patch, SOLR-8884.patch, > SOLR-8884.patch, debug.xml > > > Some of the folks > [reported|http://find.searchhub.org/document/80666f5c3b86ddda] that sometimes > explain's score can be different than the score requested by fields > parameter. Interestingly, Explain's scores would create a different ranking > than the original result list. This is something users experience, but it > cannot be re-produced deterministically. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org