[
https://issues.apache.org/jira/browse/LUCENE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683053#action_12683053
]
Michael McCandless commented on LUCENE-1522:
--------------------------------------------
{quote}
Something like that. An array of span scores is too limited; a full fledged
class would do better. Designing that class requires striking a balance
between what information we think is useful and what information Highlighter
can sanely reduce.
{quote}
Agreed, and I'm not sure about the tree structure (just floating
ideas...). It could very well be overkill.
{quote}
By proposing the tree structure, you're suggesting that
Highlighter will reverse engineer boolean matching; that sounds like a lot of
work to me.
{quote}
It wouldn't be reverse engineered: BooleanQuery/Weight/Scorer2 itself
will have returned that. Ie we would add a method to
"getSpanTree()".
{quote}
Still, the KS highlighter probably wouldn't do what you describe. The proximity
boosting accelerates as the spans approach each other, and maxes out if
they're adjacent. So "bush bush" might be prefered over "president bush",
but "bush or bush" proabably wouldn't.
{quote}
OK, it sounds like one can simply use different models to score
fragdocs and it's still an open debate how much each of these criteria
(IDF, showing surround context, being on sentence boundary, diversity
of terms) should impact the score. I agree, the "basic litmus test" I
proposed is too strong.
{quote}
bq. Lucene H1. Too many elipses, and fragments don't prefer to start on
sentence boundaries.
Thats not necessarily a property of the Highlighter, just the basic
implementations we currently supply for the pluggable classes. You can
supply a custom fragmenter and you can control the number of
fragments.
{quote}
I agree: H1 is very pluggable and one could plug in a better
fragmenter, but we don't offer such an impl in H1, and this is a case
where "out-of-the-box defaults" are very important.
> another highlighter
> -------------------
>
> Key: LUCENE-1522
> URL: https://issues.apache.org/jira/browse/LUCENE-1522
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/highlighter
> Reporter: Koji Sekiguchi
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: colored-tag-sample.png, LUCENE-1522.patch,
> LUCENE-1522.patch
>
>
> I've written this highlighter for my project to support bi-gram token stream
> (general token stream (e.g. WhitespaceTokenizer) also supported. see test
> code in patch). The idea was inherited from my previous project with my
> colleague and LUCENE-644. This approach needs highlight fields to be
> TermVector.WITH_POSITIONS_OFFSETS, but is fast and can support N-grams. This
> depends on LUCENE-1448 to get refined term offsets.
> usage:
> {code:java}
> TopDocs docs = searcher.search( query, 10 );
> Highlighter h = new Highlighter();
> FieldQuery fq = h.getFieldQuery( query );
> for( ScoreDoc scoreDoc : docs.scoreDocs ){
> // fieldName="content", fragCharSize=100, numFragments=3
> String[] fragments = h.getBestFragments( fq, reader, scoreDoc.doc,
> "content", 100, 3 );
> if( fragments != null ){
> for( String fragment : fragments )
> System.out.println( fragment );
> }
> }
> {code}
> features:
> - fast for large docs
> - supports not only whitespace-based token stream, but also "fixed size"
> N-gram (e.g. (2,2), not (1,3)) (can solve LUCENE-1489)
> - supports PhraseQuery, phrase-unit highlighting with slops
> {noformat}
> q="w1 w2"
> <b>w1 w2</b>
> ---------------
> q="w1 w2"~1
> <b>w1</b> w3 <b>w2</b> w3 <b>w1 w2</b>
> {noformat}
> - highlight fields need to be TermVector.WITH_POSITIONS_OFFSETS
> - easy to apply patch due to independent package (contrib/highlighter2)
> - uses Java 1.5
> - looks query boost to score fragments (currently doesn't see idf, but it
> should be possible)
> - pluggable FragListBuilder
> - pluggable FragmentsBuilder
> to do:
> - term positions can be unnecessary when phraseHighlight==false
> - collects performance numbers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
