On Jun 3, 2009, at 10:25 PM, Marvin Humphrey wrote:
Right now in the KS implementation, sentence boundary information is
calculated on the fly at runtime, via Highlighter_Find_Sentences().
However,
this seems wasteful, because sentence boundaries can be known at
index-time.
Perhaps we ought to be storing sentence boundary information in the
index.
Would you extend the Analysis interface to allow for custom sentence
algorithms? Could the sentences be numbered, so the final fragment has
information about *which* sentence it came from? (I could use this for
pagination.)
Perhaps if each Span were to include a reference to the original
Query object
which produced it? These would be primitives such as TermQuery and
PhraseQuery rather than compound queries like ANDQuery. Would that
reference
be enough to implement a preference for term diversity in the
excerpting algo?
There is one scenario I can think of where that *might* not work. If
someone searches for a list of keywords that includes the same keyword
twice (e.g., I sometimes copy and paste a sentence to find documents
with similar content), then there will be two TermQueries that are
identical but considered different. Maybe this won’t matter because
the duplicate term should have extra weight. I haven’t thought this
through.
And might that information come in handy for other excerpting algos?
As long as the supplied Term/PhraseQuery is the original object, and
not a clone, I think it would.
