[ http://issues.apache.org/jira/browse/LUCENE-736?page=all ]
Doron Cohen updated LUCENE-736:
-------------------------------
Attachment: sloppy_phrase.patch2.txt
res-search-orig2.log
res-search-new2.log
The change to fix case 2 was not the main performance degradation cause.
I agree with Yonik that case 2 is much more important than case 1.
So I modified to fix case 2 but not case 1.
Also extended the perf test to create also the "reversed" form of the sloppy
phrases (slop increased for reversed cases so that queries would match docs).
Cost of this fix dropped from 15% more CPU time to about 3%.
I feel ok with this.
.....Operation..........runCnt...recsPerRun...rec/s..elapsedSec....avgUsedMem....avgTotalMem
Orig.SearchSameRdr_6000......4.........6000...194.2......123.59.....8,032,732.....11,333,632
New..SearchSameRdr_6000......4.........6000...187.5......128.02.....8,172,258.....11,333,632
Attached sloppy_phrase.patch2.txt has the updated fix, including both java and
test parts. Some of the asserts in the new tests were commented out cause the
patch takes decision not to fix case 1 above.
Also attaching the updates perf test logs - res-search-orig2.log and
res-search-new2.log.
I did not compare scoring of similar cases between sloppy phrase and near spans
and Paul suggested - perhaps next week - not sure this should hold progress
with this issue.
> Sloppy Phrase Scoring Misbehavior
> ---------------------------------
>
> Key: LUCENE-736
> URL: http://issues.apache.org/jira/browse/LUCENE-736
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Reporter: Doron Cohen
> Assigned To: Doron Cohen
> Priority: Minor
> Attachments: perf-search-new.log, perf-search-orig.log,
> res-search-new2.log, res-search-orig2.log, sloppy_phrase.patch2.txt,
> sloppy_phrase_java.patch.txt, sloppy_phrase_tests.patch.txt
>
>
> This is an extension of https://issues.apache.org/jira/browse/LUCENE-697
> In addition to abnormalities Yonik pointed out in 697, there seem to be other
> issues with slopy phrase search and scoring.
> 1) A phrase with a repeated word would be detected in a document although it
> is not there.
> I.e. document = A B D C E , query = "B C B" would not find this document (as
> expected), but query "B C B"~2 would find it.
> I think that no matter how large the slop is, this document should not be a
> match.
> 2) A document containing both orders of a query, symmetrically, would score
> differently for the queru and for its reveresed form.
> I.e. document = A B C B A would score differently for queries "B C"~2 and "C
> B"~2, although it is symmetric to both.
> I will attach test cases that show both these problems and the one reported
> by Yonik in 697.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
