[
https://issues.apache.org/jira/browse/LUCENE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061480#comment-13061480
]
Mike Sokolov commented on LUCENE-2878:
--------------------------------------
bq. what do you mean by sweeping updates?
I meant adding positions to filters would be a sweeping update. But it sounds
as if the idea of rewriting differently is a better approach (certainly much
less change).
bq. For highlighting I think we should also go a two stage approach.
I think I agree. The only possible trade-off that goes the other way is in the
case where you have the positions available already during initial
search/scoring, and there is not too much turnover in the TopDocs priority
queue during hit collection. Then a Highlighter might save some time by not
re-scoring and re-iterating the positions if it accumulated them up front (even
for docs that were eventually dropped off the queue). I think it should be
possible to test out both approaches given the right API here though?
The callback idea sounds appealing, but I still think we should also consider
enabling the top-down approach: especially if this is going to run in two
passes, why not let the highlighter drive the iteration? Keep in mind that
positions consumers (like highlighters) may possibly be interested in more than
just the lowest-level positions (they may want to see phrases, eg, and
near-clauses - trying to avoid the s-word).
Another consideration is ordering. I think (?) that positions are retrieved
from the index in document order. This could be a natural order for many
cases, but score order will also be useful. I'm not sure whose responsibility
the sorting should be. Highlighters will want to be able to optimize their work
(esp for very large documents) by terminating after considering only the first
N matches, where the ordering could either be score or document-order.
I'm glad you will create a branch - this patch is getting a bit unwieldy. I
think the PosHighlighter code should probably (?) end up as test code only - I
guess we'll see. It seems like we could get further faster using the existing
Highlighter, with a positions-based TokenStream; I'll post a patch once the
branch is in place.
> Allow Scorer to expose positions and payloads aka. nuke spans
> --------------------------------------------------------------
>
> Key: LUCENE-2878
> URL: https://issues.apache.org/jira/browse/LUCENE-2878
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/search
> Affects Versions: Bulk Postings branch
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Labels: gsoc2011, lucene-gsoc-11, mentor
> Attachments: LUCENE-2878-OR.patch, LUCENE-2878.patch,
> LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
> LUCENE-2878_trunk.patch, LUCENE-2878_trunk.patch, PosHighlighter.patch,
> PosHighlighter.patch
>
>
> Currently we have two somewhat separate types of queries, the one which can
> make use of positions (mainly spans) and payloads (spans). Yet Span*Query
> doesn't really do scoring comparable to what other queries do and at the end
> of the day they are duplicating lot of code all over lucene. Span*Queries are
> also limited to other Span*Query instances such that you can not use a
> TermQuery or a BooleanQuery with SpanNear or anthing like that.
> Beside of the Span*Query limitation other queries lacking a quiet interesting
> feature since they can not score based on term proximity since scores doesn't
> expose any positional information. All those problems bugged me for a while
> now so I stared working on that using the bulkpostings API. I would have done
> that first cut on trunk but TermScorer is working on BlockReader that do not
> expose positions while the one in this branch does. I started adding a new
> Positions class which users can pull from a scorer, to prevent unnecessary
> positions enums I added ScorerContext#needsPositions and eventually
> Scorere#needsPayloads to create the corresponding enum on demand. Yet,
> currently only TermQuery / TermScorer implements this API and other simply
> return null instead.
> To show that the API really works and our BulkPostings work fine too with
> positions I cut over TermSpanQuery to use a TermScorer under the hood and
> nuked TermSpans entirely. A nice sideeffect of this was that the Position
> BulkReading implementation got some exercise which now :) work all with
> positions while Payloads for bulkreading are kind of experimental in the
> patch and those only work with Standard codec.
> So all spans now work on top of TermScorer ( I truly hate spans since today )
> including the ones that need Payloads (StandardCodec ONLY)!! I didn't bother
> to implement the other codecs yet since I want to get feedback on the API and
> on this first cut before I go one with it. I will upload the corresponding
> patch in a minute.
> I also had to cut over SpanQuery.getSpans(IR) to
> SpanQuery.getSpans(AtomicReaderContext) which I should probably do on trunk
> first but after that pain today I need a break first :).
> The patch passes all core tests
> (org.apache.lucene.search.highlight.HighlighterTest still fails but I didn't
> look into the MemoryIndex BulkPostings API yet)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
