Let me give some background on the problem behind my question.
Our index contains many fields (title, body, date, city, etc). Most queries
search all fields, but for best performance, we create an additional
'contents' field that contains all terms from all fields so that only one
field needs to be searched. Some fields, like title and city, are boosted by
a factor of 5. In order to make term boosting work, we create an additional
field 'boost' that contains all the terms from the boosted fields (title,
city).
Then, at search time, a query for "petroleum engineer" gets rewritten to:
(+contents:petroleum +contents:engineer) (+boost:petroleum +boost:engineer).
Note that the two clauses are OR'd so that a term that exists in both fields
will get a higher weight in the 'boost' field. This works quite well at
boosting documents with terms that exist in the boosted fields. However, it
doesn't work properly if excluded terms are added, for example:
(+contents:petroleum +contents:engineer -contents:drilling)
(+boost:petroleum +boost:engineer -boost:drilling)
If a document contains the term 'drilling' in the 'body' field, but not in
the 'title' or 'city' field, a false hit occurs.
Enter payloads and 'BoostingTermQuery'. At indexing time, as terms are added
to the 'contents' field, they are assigned a payload (value=5) if the term
also exists in one of the boosted fields. The 'scorePayload' method in our
Similarity class returns the payload value as a score. The query no longer
contains the 'boost' fields and is simply:
+contents:petroleum +contents:engineer -contents:drilling
The goal is to make the payload technique behavior similar to the 'boost'
field technique. The problem is that relevance scores of the top hits are
sometimes quite different. The reason is that the IDF values for a given
term in the 'boost' field is often much higher than the same term in the
'contents' field. This makes sense because the 'boost' field contains a
fairly small subset of the 'contents' field. Even with a payload of '5', a
low IDF in the 'contents' field usually erases the effect of the payload.
I have found a fairly simple (albeit inelegant) solution that seems to work.
The 'boost' field is still created as before, but it is only used to compute
IDF values for the weight class 'BoostingTermQuery.BoostingTermWeight. I had
to make this class 'public' so that I could override the IDF value as
follows:
public class MNSBoostingTermQuery extends BoostingTermQuery {
public MNSBoostingTermQuery(Term term) {
super(term);
}
protected class MNSBoostingTermWeight extends
BoostingTermQuery.BoostingTermWeight {
public MNSBoostingTermWeight(BoostingTermQuery query, Searcher searcher)
throws IOException {
super(query, searcher);
java.util.HashSet<Term> newTerms = new java.util.HashSet<Term>();
// Recompute IDF based on 'boost' field
Iterator i = terms.iterator();
Term term=null;
while (i.hasNext()) {
term = (Term)i.next();
newTerms.add(new Term("boost", term.text()));
}
this.idf = this.query.getSimilarity(searcher).idf(newTerms, searcher);
}
}
}
Any thoughts about a better implementation are welcome.
Peter
On Thu, Nov 6, 2008 at 8:00 AM, Grant Ingersoll <[EMAIL PROTECTED]> wrote:
> Not sure, but it sounds like you are interested in a higher level Query,
> kind of like the BooleanQuery, but then part of it sounds like it is per
> document, right? Is it that you want to deal with multiple payloads in a
> document, or multiple BTQs in a bigger query?
>
> On Nov 4, 2008, at 9:42 AM, Peter Keegan wrote:
>
> I'm using BoostingTermQuery to boost the score of documents with terms
>> containing payloads (boost value > 1). I'd like to change the scoring
>> behavior such that if a query contains multiple BoostingTermQuery terms
>> (either required or optional), documents containing more matching terms
>> with
>> payloads always score higher than documents with fewer terms with
>> payloads.
>> Currently, if one of the terms has a high IDF weight and contains a
>> boosting
>> payload but no payloads on other matching terms, it may score higher than
>> docs with other matching terms with payloads and lower IDF.
>>
>> I think what I need is a way to increase the weight of a matching term in
>> BoostingSpanScorer.score() if 'payloadsSeen > 0', but I don't see how to
>> do
>> this. Any suggestions?
>>
>> Thanks,
>> Peter
>>
>
> --------------------------
> Grant Ingersoll
>
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
