I've discovered another flaw in using this technique:
(+contents:petroleum +contents:engineer +contents:refinery)
(+boost:petroleum +boost:engineer +boost:refinery)
It's possible that the first clause will produce a matching doc and none of
the terms in the second clause are used to score that doc. Yet another
reason to use BoostingTermQuery.
Peter
On Thu, Nov 6, 2008 at 1:08 PM, Peter Keegan <[EMAIL PROTECTED]> wrote:
> Let me give some background on the problem behind my question.
>
> Our index contains many fields (title, body, date, city, etc). Most queries
> search all fields, but for best performance, we create an additional
> 'contents' field that contains all terms from all fields so that only one
> field needs to be searched. Some fields, like title and city, are boosted by
> a factor of 5. In order to make term boosting work, we create an additional
> field 'boost' that contains all the terms from the boosted fields (title,
> city).
>
> Then, at search time, a query for "petroleum engineer" gets rewritten to:
> (+contents:petroleum +contents:engineer) (+boost:petroleum +boost:engineer).
> Note that the two clauses are OR'd so that a term that exists in both fields
> will get a higher weight in the 'boost' field. This works quite well at
> boosting documents with terms that exist in the boosted fields. However, it
> doesn't work properly if excluded terms are added, for example:
>
> (+contents:petroleum +contents:engineer -contents:drilling)
> (+boost:petroleum +boost:engineer -boost:drilling)
>
> If a document contains the term 'drilling' in the 'body' field, but not in
> the 'title' or 'city' field, a false hit occurs.
>
> Enter payloads and 'BoostingTermQuery'. At indexing time, as terms are
> added to the 'contents' field, they are assigned a payload (value=5) if the
> term also exists in one of the boosted fields. The 'scorePayload' method in
> our Similarity class returns the payload value as a score. The query no
> longer contains the 'boost' fields and is simply:
>
> +contents:petroleum +contents:engineer -contents:drilling
>
> The goal is to make the payload technique behavior similar to the 'boost'
> field technique. The problem is that relevance scores of the top hits are
> sometimes quite different. The reason is that the IDF values for a given
> term in the 'boost' field is often much higher than the same term in the
> 'contents' field. This makes sense because the 'boost' field contains a
> fairly small subset of the 'contents' field. Even with a payload of '5', a
> low IDF in the 'contents' field usually erases the effect of the payload.
>
> I have found a fairly simple (albeit inelegant) solution that seems to
> work. The 'boost' field is still created as before, but it is only used to
> compute IDF values for the weight class
> 'BoostingTermQuery.BoostingTermWeight. I had to make this class 'public' so
> that I could override the IDF value as follows:
>
> public class MNSBoostingTermQuery extends BoostingTermQuery {
> public MNSBoostingTermQuery(Term term) {
> super(term);
> }
> protected class MNSBoostingTermWeight extends
> BoostingTermQuery.BoostingTermWeight {
> public MNSBoostingTermWeight(BoostingTermQuery query, Searcher
> searcher) throws IOException {
> super(query, searcher);
> java.util.HashSet<Term> newTerms = new java.util.HashSet<Term>();
> // Recompute IDF based on 'boost' field
> Iterator i = terms.iterator();
> Term term=null;
> while (i.hasNext()) {
> term = (Term)i.next();
> newTerms.add(new Term("boost", term.text()));
> }
> this.idf = this.query.getSimilarity(searcher).idf(newTerms,
> searcher);
> }
> }
> }
>
> Any thoughts about a better implementation are welcome.
>
> Peter
>
>
>
>
>
> On Thu, Nov 6, 2008 at 8:00 AM, Grant Ingersoll <[EMAIL PROTECTED]>wrote:
>
>> Not sure, but it sounds like you are interested in a higher level Query,
>> kind of like the BooleanQuery, but then part of it sounds like it is per
>> document, right? Is it that you want to deal with multiple payloads in a
>> document, or multiple BTQs in a bigger query?
>>
>> On Nov 4, 2008, at 9:42 AM, Peter Keegan wrote:
>>
>> I'm using BoostingTermQuery to boost the score of documents with terms
>>> containing payloads (boost value > 1). I'd like to change the scoring
>>> behavior such that if a query contains multiple BoostingTermQuery terms
>>> (either required or optional), documents containing more matching terms
>>> with
>>> payloads always score higher than documents with fewer terms with
>>> payloads.
>>> Currently, if one of the terms has a high IDF weight and contains a
>>> boosting
>>> payload but no payloads on other matching terms, it may score higher than
>>> docs with other matching terms with payloads and lower IDF.
>>>
>>> I think what I need is a way to increase the weight of a matching term in
>>> BoostingSpanScorer.score() if 'payloadsSeen > 0', but I don't see how to
>>> do
>>> this. Any suggestions?
>>>
>>> Thanks,
>>> Peter
>>>
>>
>> --------------------------
>> Grant Ingersoll
>>
>>
>> Lucene Helpful Hints:
>> http://wiki.apache.org/lucene-java/BasicsOfPerformance
>> http://wiki.apache.org/lucene-java/LuceneFAQ
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>>
>
