[
https://issues.apache.org/jira/browse/LUCENE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411425#comment-13411425
]
Simon Willnauer commented on LUCENE-2878:
-----------------------------------------
hey alan,
I merged the branch up with trunk and applied you patch (latest-1). Your
changes to PhraseQuery are tricky. The SloppyPhraseScorer now uses the same
DocsAndPosition enums as the PosIterator in there. That is unfortunately not
how it should be. If you pull a PosIterator from a scorer there should not be
any side-effect on the score or the iterator if you advance one or the other.
Currently I see a failure in PosHighlighterTest:
{noformat}
[junit4:junit4] Suite: org.apache.lucene.search.poshighlight.PosHighlighterTest
[junit4:junit4] FAILURE 0.22s J0 | PosHighlighterTest.testSloppyPhraseQuery
[junit4:junit4] > Throwable #1: java.lang.AssertionError: nextPosition() was
called too many times (more than freq() times) posPendingCount=-1
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([C966081DA1EFC306:32134412A4F88738]:0)
[junit4:junit4] > at
org.apache.lucene.codecs.lucene40.Lucene40PostingsReader$SegmentFullPositionsEnum.nextPosition(Lucene40PostingsReader.java:1127)
[junit4:junit4] > at
org.apache.lucene.search.PhrasePositions.nextPosition(PhrasePositions.java:76)
[junit4:junit4] > at
org.apache.lucene.search.PhrasePositions.firstPosition(PhrasePositions.java:65)
[junit4:junit4] > at
org.apache.lucene.search.SloppyPhraseScorer.initSimple(SloppyPhraseScorer.java:230)
[junit4:junit4] > at
org.apache.lucene.search.SloppyPhraseScorer.initPhrasePositions(SloppyPhraseScorer.java:218)
[junit4:junit4] > at
org.apache.lucene.search.SloppyPhraseScorer.access$800(SloppyPhraseScorer.java:28)
[junit4:junit4] > at
org.apache.lucene.search.SloppyPhraseScorer$SloppyPhrasePositionIntervalIterator.advanceTo(SloppyPhraseScorer.java:533)
[junit4:junit4] > at
org.apache.lucene.search.poshighlight.PosCollector.collect(PosCollector.java:53)
[junit4:junit4] > at org.apache.lucene.search.Scorer.score(Scorer.java:62)
[junit4:junit4] > at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:574)
[junit4:junit4] > at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:287)
[junit4:junit4] > at
org.apache.lucene.search.poshighlight.PosHighlighterTest.doSearch(PosHighlighterTest.java:161)
[junit4:junit4] > at
org.apache.lucene.search.poshighlight.PosHighlighterTest.doSearch(PosHighlighterTest.java:147)
[junit4:junit4] > at
org.apache.lucene.search.poshighlight.PosHighlighterTest.testSloppyPhraseQuery(PosHighlighterTest.java:378)
...
[junit4:junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=PosHighlighterTest -Dtests.method=testSloppyPhraseQuery
-Dtests.seed=C966081DA1EFC306 -Dtests.slow=true -Dtests.locale=sr
-Dtests.timezone=Africa/Harare -Dtests.file.encoding=UTF-8
[junit4:junit4] 2>
[junit4:junit4] > (@AfterClass output)
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40: {},
sim=RandomSimilarityProvider(queryNorm=true,coord=false): {f=DFR
I(ne)B3(800.0)}, locale=sr, timezone=Africa/Harare
[junit4:junit4] 2> NOTE: Linux 2.6.38-15-generic amd64/Sun Microsystems Inc.
1.6.0_26 (64-bit)/cpus=12,threads=1,free=327745216,total=379322368
[junit4:junit4] 2> NOTE: All tests run in this JVM: [PosHighlighterTest]
[junit4:junit4] 2>
{noformat}
this makes the entire sloppy case very tricky though. Even further I don't
think sloppy phrase is really correct and if I recall correctly there some
issue with it that haven't been resolved for years now. I am not sure how we
should proceed with that one. I will need to think about that further.
bq. Next step is to try and reimplement the various SpanQuery tests using the
position filter queries in their place.
please go ahead!
> Allow Scorer to expose positions and payloads aka. nuke spans
> --------------------------------------------------------------
>
> Key: LUCENE-2878
> URL: https://issues.apache.org/jira/browse/LUCENE-2878
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Affects Versions: Positions Branch
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Labels: gsoc2011, gsoc2012, lucene-gsoc-11, lucene-gsoc-12,
> mentor
> Fix For: Positions Branch
>
> Attachments: LUCENE-2878-OR.patch, LUCENE-2878.patch,
> LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
> LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
> LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
> LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
> LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878_trunk.patch,
> LUCENE-2878_trunk.patch, PosHighlighter.patch, PosHighlighter.patch
>
>
> Currently we have two somewhat separate types of queries, the one which can
> make use of positions (mainly spans) and payloads (spans). Yet Span*Query
> doesn't really do scoring comparable to what other queries do and at the end
> of the day they are duplicating lot of code all over lucene. Span*Queries are
> also limited to other Span*Query instances such that you can not use a
> TermQuery or a BooleanQuery with SpanNear or anthing like that.
> Beside of the Span*Query limitation other queries lacking a quiet interesting
> feature since they can not score based on term proximity since scores doesn't
> expose any positional information. All those problems bugged me for a while
> now so I stared working on that using the bulkpostings API. I would have done
> that first cut on trunk but TermScorer is working on BlockReader that do not
> expose positions while the one in this branch does. I started adding a new
> Positions class which users can pull from a scorer, to prevent unnecessary
> positions enums I added ScorerContext#needsPositions and eventually
> Scorere#needsPayloads to create the corresponding enum on demand. Yet,
> currently only TermQuery / TermScorer implements this API and other simply
> return null instead.
> To show that the API really works and our BulkPostings work fine too with
> positions I cut over TermSpanQuery to use a TermScorer under the hood and
> nuked TermSpans entirely. A nice sideeffect of this was that the Position
> BulkReading implementation got some exercise which now :) work all with
> positions while Payloads for bulkreading are kind of experimental in the
> patch and those only work with Standard codec.
> So all spans now work on top of TermScorer ( I truly hate spans since today )
> including the ones that need Payloads (StandardCodec ONLY)!! I didn't bother
> to implement the other codecs yet since I want to get feedback on the API and
> on this first cut before I go one with it. I will upload the corresponding
> patch in a minute.
> I also had to cut over SpanQuery.getSpans(IR) to
> SpanQuery.getSpans(AtomicReaderContext) which I should probably do on trunk
> first but after that pain today I need a break first :).
> The patch passes all core tests
> (org.apache.lucene.search.highlight.HighlighterTest still fails but I didn't
> look into the MemoryIndex BulkPostings API yet)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
