[ 
https://issues.apache.org/jira/browse/LUCENE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man updated LUCENE-7132:
-----------------------------
    Attachment: LUCENE-7132.patch


W/o really having firm idea what's going on here i spent some time reviewing 
the tests that Ahmet posted, and then started poking at things with a stick -- 
mainly to see if I could figure out the missing piece of the puzzle that caused 
things to fail in "testRajeshData()" but not in any of our existing randomized 
tests.  

I feel down the rabit hole a bit looking into this, and I still have no 
concrete idea what the underlying problem is, but i have a few uneducated 
guesses...

* the problem seems to only affect BooleanQueries where a "coord" factor is 
involved
** adding {{setDisableCoord(true)}} on the queries in testRajeshData seems to 
make all seeds pass
** so obviously reproducing this bug requires a Similarity that has a 
non-trivila coord function (like ClassicSimilaity in Ahmet's TestExplain)
* the _score_ values are inconsistent between randomized runs of 
testRajeshData, while the _explain_ values stay the same
** even though the test data & similarity are fixed, testRajeshData pass fine 
with some seeds, while others fail
** when testRajeshData does fail, it's because the scores have changed compared 
to the values produced in previous passing runs
** {color:red}best guess: something about the MergePolicy and the way docs are 
co-located in diff segments is triggering a pathological code path in some 
BooleanWeight/Scorer optimization code?{color}
*** perhaps because a segment may not even contain one of the optional terms, 
so it's Scorer is null?
** if it's not the MergePolicy, then perhaps something about the codec used and 
the term stats?

Things i've added/revised in this new version of the patch...

* added some more asserts to TestExplain.testRajeshData to help demonstrate 
that it's the score that has changed between successful vs failing runs
* fixed a bug in TestExplain.testExplainScoreEquality that was causing false 
failures
** {{randomSimpleString(random())}} can produce the empty string, which was 
causing the test to not match the numDocs it was expecting since a whitespace 
based analyzer is being used
** NOTE: these false failures where the only case i've ever seen 
TestExplain.testExplainScoreEquality fail
* added a TestBaseExplanationTestCase to the test-framework
** I had some initial concerns that maybe some old changes/refactorings to 
BaseExplanationTestCase had completely eliminated the checks that were suppose 
to be done in that test, so i added this class to ensure it would fail as 
expected when an Explanation didn't match scores
* added more coord randomization in TestSimpleExplanations
** even w/these changes, i've never seen these tests fail
* added a new TestSimpleExplanationsWithHeavyIDF
** this was one of the first things i tried, based on Ahmet's concerns about 
{{IDFStats.normalize}}
** since TestExplain.testRajeshData was failing so consistently, but we've 
never seen failures like this from any of of the existing 
BaseExplanationTestCase, i speculated that maybe IDF stats or index size were a 
big factor, and tried to bang out a subclass of TestSimpleExplanations that 
added a lot of (ignored) docs with the same terms to make the doc freq stats 
more interesting for the terms being searched on
** i've never seen this test fail
** some refactoring in BaseExplanationTestCase was needed to write this test

----

As things stand now with the current patch, here are some test seeds for 
TestExplain that pass for me...

{noformat}
ant test  -Dtestcase=TestExplain -Dtests.method=testRajeshData 
-Dtests.seed=11679FC4F397A674 -Dtests.slow=true -Dtests.locale=th 
-Dtests.timezone=Etc/GMT+1 -Dtests.asserts=true -Dtests.file.encoding=UTF-8 
-Dtests.verbose=true
...
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): 
{id=PostingsFormat(name=SimpleText), body=PostingsFormat(name=Memory doPackFST= 
false)}, docValues:{}, maxPointsInLeafNode=604, 
maxMBSortInHeap=6.961830329642295, sim=ClassicSimilarity, locale=th, 
timezone=Etc/GMT+1
{noformat}
{noformat}
ant test -Dtestcase=TestExplain -Dtests.verbose=true 
-Dtests.seed=610EF558241D1898 -Dtests.slow=true -Dtests.locale=th 
-Dtests.timezone=Etc/GMT+1 -Dtests.asserts=true -Dtests.file.encoding=UTF-8 
-Dtests.verbose=true
...
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): 
{id=PostingsFormat(name=Memory doPackFST= true), body=Lucene50(blocksize=128)}, 
docValues:{}, maxPointsInLeafNode=738, maxMBSortInHeap=5.910307356558968, 
sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=th, timezone=Etc/GMT+1
{noformat}

And here are some examples of failures...


fails...

{noformat}
ant test  -Dtestcase=TestExplain -Dtests.method=testRajeshData 
-Dtests.seed=A2E2DCFC7459241E -Dtests.slow=true -Dtests.locale=ar-EG 
-Dtests.timezone=Africa/Asmara -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII -Dtests.verbose=true

   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestExplain 
-Dtests.method=testRajeshData -Dtests.seed=A2E2DCFC7459241E -Dtests.slow=true 
-Dtests.locale=ar-EG -Dtests.timezone=Africa/Asmara -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 1.09s | TestExplain.testRajeshData <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: o-365 score 
expected:<0.7475659251213074> but was:<0.2491886466741562>
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([A2E2DCFC7459241E:C3C4FAF539C031CF]:0)
   [junit4]    >        at 
org.apache.lucene.search.TestExplain.testRajeshData(TestExplain.java:210)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, 
sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=ar-EG, 
timezone=Africa/Asmara
   [junit4]   2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation 
1.8.0_74 (64-bit)/cpus=4,threads=1,free=236148720,total=248512512
{noformat}

{noformat}
ant test  -Dtestcase=TestExplain -Dtests.method=testRajeshData 
-Dtests.seed=92DC100B28B70C2C -Dtests.slow=true -Dtests.locale=he 
-Dtests.timezone=America/Guyana -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8  -Dtests.verbose=true

   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestExplain 
-Dtests.method=testRajeshData -Dtests.seed=92DC100B28B70C2C -Dtests.slow=true 
-Dtests.locale=he -Dtests.timezone=America/Guyana -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.03s | TestExplain.testRajeshData <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: m-o-365 score 
expected:<1.2357021570205688> but was:<0.41190072894096375>
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([92DC100B28B70C2C:F3FA3602652E19FD]:0)
   [junit4]    >        at 
org.apache.lucene.search.TestExplain.testRajeshData(TestExplain.java:207)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, 
locale=he, timezone=America/Guyana
   [junit4]   2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation 
1.8.0_74 (64-bit)/cpus=4,threads=1,free=173898752,total=249561088
   [junit4]   2> NOTE: All tests run in this JVM: [TestExplain]
{noformat}

My refactoring of BaseExplanationTestCase also seems to have somehow 
introduced/tickled a stranage bug in the test framework that trips our 
SecurityManager settings -- note that the seed below causes an odd 
AccessControlException at the suite level *after* all TestSimpleExplanations 
test methods have finished successfully...

{noformat}
ant test  -Dtestcase=TestSimpleExplanations -Dtests.seed=74B719CE50C8168A 
-Dtests.slow=true -Dtests.locale=sr-Latn -Dtests.timezone=America/St_Vincent 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8

   ...
   [junit4] OK      0.01s | TestSimpleExplanations.testBQ14
   [junit4] OK      0.01s | TestSimpleExplanations.testBQ1
   [junit4] OK      0.03s | TestSimpleExplanations.testBQ20
   [junit4] OK      0.03s | TestSimpleExplanations.testBQ4
   [junit4] OK      0.01s | TestSimpleExplanations.testP2
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60), 
sim=RandomSimilarity(queryNorm=false,coord=no): {field=DFR I(n)B2, alt=DFR 
I(ne)B1}, locale=sr-Latn, timezone=America/St_Vincent
   [junit4]   2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation 
1.8.0_74 (64-bit)/cpus=4,threads=1,free=303553832,total=329777152
   [junit4]   2> NOTE: All tests run in this JVM: [TestSimpleExplanations]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestSimpleExplanations -Dtests.seed=74B719CE50C8168A 
-Dtests.slow=true -Dtests.locale=sr-Latn -Dtests.timezone=America/St_Vincent 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s | TestSimpleExplanations (suite) <<<
   [junit4]    > Throwable #1: java.security.AccessControlException: access 
denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.nio.fs")
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([74B719CE50C8168A]:0)
   [junit4]    >        at 
java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
   [junit4]    >        at 
java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    >        at 
java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    >        at 
java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1564)
   [junit4]    >        at java.lang.Class.checkPackageAccess(Class.java:2372)
   [junit4]    >        at java.lang.Class.checkMemberAccess(Class.java:2351)
   [junit4]    >        at java.lang.Class.getDeclaredFields(Class.java:1915)
   [junit4]    >        at java.security.AccessController.doPrivileged(Native 
Method)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [1/1 (1!)] in 2.50s, 68 tests, 1 error <<< FAILURES!
{noformat}


> ScoreDoc.score() returns a different value than that of Explanation's
> ---------------------------------------------------------------------
>
>                 Key: LUCENE-7132
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7132
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/search
>    Affects Versions: 5.5
>            Reporter: Ahmet Arslan
>            Assignee: Steve Rowe
>         Attachments: LUCENE-7132.patch, LUCENE-7132.patch, SOLR-8884.patch, 
> SOLR-8884.patch, debug.xml
>
>
> Some of the folks 
> [reported|http://find.searchhub.org/document/80666f5c3b86ddda] that sometimes 
> explain's score can be different than the score requested by fields 
> parameter. Interestingly, Explain's scores would create a different ranking 
> than the original result list. This is something users experience, but it 
> cannot be re-produced deterministically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to