[
https://issues.apache.org/jira/browse/LUCENE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated LUCENE-7132:
-----------------------------
Attachment: LUCENE-7132.patch
W/o really having firm idea what's going on here i spent some time reviewing
the tests that Ahmet posted, and then started poking at things with a stick --
mainly to see if I could figure out the missing piece of the puzzle that caused
things to fail in "testRajeshData()" but not in any of our existing randomized
tests.
I feel down the rabit hole a bit looking into this, and I still have no
concrete idea what the underlying problem is, but i have a few uneducated
guesses...
* the problem seems to only affect BooleanQueries where a "coord" factor is
involved
** adding {{setDisableCoord(true)}} on the queries in testRajeshData seems to
make all seeds pass
** so obviously reproducing this bug requires a Similarity that has a
non-trivila coord function (like ClassicSimilaity in Ahmet's TestExplain)
* the _score_ values are inconsistent between randomized runs of
testRajeshData, while the _explain_ values stay the same
** even though the test data & similarity are fixed, testRajeshData pass fine
with some seeds, while others fail
** when testRajeshData does fail, it's because the scores have changed compared
to the values produced in previous passing runs
** {color:red}best guess: something about the MergePolicy and the way docs are
co-located in diff segments is triggering a pathological code path in some
BooleanWeight/Scorer optimization code?{color}
*** perhaps because a segment may not even contain one of the optional terms,
so it's Scorer is null?
** if it's not the MergePolicy, then perhaps something about the codec used and
the term stats?
Things i've added/revised in this new version of the patch...
* added some more asserts to TestExplain.testRajeshData to help demonstrate
that it's the score that has changed between successful vs failing runs
* fixed a bug in TestExplain.testExplainScoreEquality that was causing false
failures
** {{randomSimpleString(random())}} can produce the empty string, which was
causing the test to not match the numDocs it was expecting since a whitespace
based analyzer is being used
** NOTE: these false failures where the only case i've ever seen
TestExplain.testExplainScoreEquality fail
* added a TestBaseExplanationTestCase to the test-framework
** I had some initial concerns that maybe some old changes/refactorings to
BaseExplanationTestCase had completely eliminated the checks that were suppose
to be done in that test, so i added this class to ensure it would fail as
expected when an Explanation didn't match scores
* added more coord randomization in TestSimpleExplanations
** even w/these changes, i've never seen these tests fail
* added a new TestSimpleExplanationsWithHeavyIDF
** this was one of the first things i tried, based on Ahmet's concerns about
{{IDFStats.normalize}}
** since TestExplain.testRajeshData was failing so consistently, but we've
never seen failures like this from any of of the existing
BaseExplanationTestCase, i speculated that maybe IDF stats or index size were a
big factor, and tried to bang out a subclass of TestSimpleExplanations that
added a lot of (ignored) docs with the same terms to make the doc freq stats
more interesting for the terms being searched on
** i've never seen this test fail
** some refactoring in BaseExplanationTestCase was needed to write this test
----
As things stand now with the current patch, here are some test seeds for
TestExplain that pass for me...
{noformat}
ant test -Dtestcase=TestExplain -Dtests.method=testRajeshData
-Dtests.seed=11679FC4F397A674 -Dtests.slow=true -Dtests.locale=th
-Dtests.timezone=Etc/GMT+1 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
-Dtests.verbose=true
...
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60):
{id=PostingsFormat(name=SimpleText), body=PostingsFormat(name=Memory doPackFST=
false)}, docValues:{}, maxPointsInLeafNode=604,
maxMBSortInHeap=6.961830329642295, sim=ClassicSimilarity, locale=th,
timezone=Etc/GMT+1
{noformat}
{noformat}
ant test -Dtestcase=TestExplain -Dtests.verbose=true
-Dtests.seed=610EF558241D1898 -Dtests.slow=true -Dtests.locale=th
-Dtests.timezone=Etc/GMT+1 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
-Dtests.verbose=true
...
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60):
{id=PostingsFormat(name=Memory doPackFST= true), body=Lucene50(blocksize=128)},
docValues:{}, maxPointsInLeafNode=738, maxMBSortInHeap=5.910307356558968,
sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=th, timezone=Etc/GMT+1
{noformat}
And here are some examples of failures...
fails...
{noformat}
ant test -Dtestcase=TestExplain -Dtests.method=testRajeshData
-Dtests.seed=A2E2DCFC7459241E -Dtests.slow=true -Dtests.locale=ar-EG
-Dtests.timezone=Africa/Asmara -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII -Dtests.verbose=true
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestExplain
-Dtests.method=testRajeshData -Dtests.seed=A2E2DCFC7459241E -Dtests.slow=true
-Dtests.locale=ar-EG -Dtests.timezone=Africa/Asmara -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII
[junit4] FAILURE 1.09s | TestExplain.testRajeshData <<<
[junit4] > Throwable #1: java.lang.AssertionError: o-365 score
expected:<0.7475659251213074> but was:<0.2491886466741562>
[junit4] > at
__randomizedtesting.SeedInfo.seed([A2E2DCFC7459241E:C3C4FAF539C031CF]:0)
[junit4] > at
org.apache.lucene.search.TestExplain.testRajeshData(TestExplain.java:210)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=CheapBastard,
sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=ar-EG,
timezone=Africa/Asmara
[junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation
1.8.0_74 (64-bit)/cpus=4,threads=1,free=236148720,total=248512512
{noformat}
{noformat}
ant test -Dtestcase=TestExplain -Dtests.method=testRajeshData
-Dtests.seed=92DC100B28B70C2C -Dtests.slow=true -Dtests.locale=he
-Dtests.timezone=America/Guyana -Dtests.asserts=true
-Dtests.file.encoding=UTF-8 -Dtests.verbose=true
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestExplain
-Dtests.method=testRajeshData -Dtests.seed=92DC100B28B70C2C -Dtests.slow=true
-Dtests.locale=he -Dtests.timezone=America/Guyana -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 1.03s | TestExplain.testRajeshData <<<
[junit4] > Throwable #1: java.lang.AssertionError: m-o-365 score
expected:<1.2357021570205688> but was:<0.41190072894096375>
[junit4] > at
__randomizedtesting.SeedInfo.seed([92DC100B28B70C2C:F3FA3602652E19FD]:0)
[junit4] > at
org.apache.lucene.search.TestExplain.testRajeshData(TestExplain.java:207)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity,
locale=he, timezone=America/Guyana
[junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation
1.8.0_74 (64-bit)/cpus=4,threads=1,free=173898752,total=249561088
[junit4] 2> NOTE: All tests run in this JVM: [TestExplain]
{noformat}
My refactoring of BaseExplanationTestCase also seems to have somehow
introduced/tickled a stranage bug in the test framework that trips our
SecurityManager settings -- note that the seed below causes an odd
AccessControlException at the suite level *after* all TestSimpleExplanations
test methods have finished successfully...
{noformat}
ant test -Dtestcase=TestSimpleExplanations -Dtests.seed=74B719CE50C8168A
-Dtests.slow=true -Dtests.locale=sr-Latn -Dtests.timezone=America/St_Vincent
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
...
[junit4] OK 0.01s | TestSimpleExplanations.testBQ14
[junit4] OK 0.01s | TestSimpleExplanations.testBQ1
[junit4] OK 0.03s | TestSimpleExplanations.testBQ20
[junit4] OK 0.03s | TestSimpleExplanations.testBQ4
[junit4] OK 0.01s | TestSimpleExplanations.testP2
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60),
sim=RandomSimilarity(queryNorm=false,coord=no): {field=DFR I(n)B2, alt=DFR
I(ne)B1}, locale=sr-Latn, timezone=America/St_Vincent
[junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation
1.8.0_74 (64-bit)/cpus=4,threads=1,free=303553832,total=329777152
[junit4] 2> NOTE: All tests run in this JVM: [TestSimpleExplanations]
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestSimpleExplanations -Dtests.seed=74B719CE50C8168A
-Dtests.slow=true -Dtests.locale=sr-Latn -Dtests.timezone=America/St_Vincent
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s | TestSimpleExplanations (suite) <<<
[junit4] > Throwable #1: java.security.AccessControlException: access
denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.nio.fs")
[junit4] > at
__randomizedtesting.SeedInfo.seed([74B719CE50C8168A]:0)
[junit4] > at
java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
[junit4] > at
java.security.AccessController.checkPermission(AccessController.java:884)
[junit4] > at
java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[junit4] > at
java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1564)
[junit4] > at java.lang.Class.checkPackageAccess(Class.java:2372)
[junit4] > at java.lang.Class.checkMemberAccess(Class.java:2351)
[junit4] > at java.lang.Class.getDeclaredFields(Class.java:1915)
[junit4] > at java.security.AccessController.doPrivileged(Native
Method)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] Completed [1/1 (1!)] in 2.50s, 68 tests, 1 error <<< FAILURES!
{noformat}
> ScoreDoc.score() returns a different value than that of Explanation's
> ---------------------------------------------------------------------
>
> Key: LUCENE-7132
> URL: https://issues.apache.org/jira/browse/LUCENE-7132
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
> Affects Versions: 5.5
> Reporter: Ahmet Arslan
> Assignee: Steve Rowe
> Attachments: LUCENE-7132.patch, LUCENE-7132.patch, SOLR-8884.patch,
> SOLR-8884.patch, debug.xml
>
>
> Some of the folks
> [reported|http://find.searchhub.org/document/80666f5c3b86ddda] that sometimes
> explain's score can be different than the score requested by fields
> parameter. Interestingly, Explain's scores would create a different ranking
> than the original result list. This is something users experience, but it
> cannot be re-produced deterministically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
