Hi Chris,
Can we pass a different query object for searching and a different one
to the highlighter? I am not sure of that.
In any case, based on Mark's suggestion I modified the
QueryTermsExtractor class and filtered the query terms by the fieldName.
Attached is the modified file.
Thanks,
Harini
Chris Hostetter wrote:
I don't know what your application is, and I have no experience with the
Highlighter code, so forgive me if this is a silly suggestion:
It looks like you are building a query up programaticaly, which
contains some words to search on, and some other stuff that's mainly
being used to "filter" the results (i'll avoid my usual rant about
people underutilizing Filters). So why not pass the Higherlighter just
the portion of the Query that you acctaully want to contribute to the
highlighting? In this query...
: >> +DocumentType:news
: >> +(CompanyId:10 CompanyId:20 CompanyId:30 CompanyId:40)
: >> +FilingDate:[20041201 TO 20051201]
: >> +(Content:"cost saving" Content:"cost savings"
: >>Content:outsource
: >>Content:outsources Content:downsize
: >>Content:downsizes
: >>Content:restructuring Content:restructure)
...just give the highlighter...
(Content:"cost saving" Content:"cost savings"
Content:outsource
Content:outsources Content:downsize
Content:downsizes
Content:restructuring Content:restructure)
: Date: Thu, 01 Dec 2005 10:38:41 +0530
: From: Harini Raghavan <[EMAIL PROTECTED]>
: Reply-To: java-user@lucene.apache.org
: To: java-user@lucene.apache.org
: Subject: Re: how to control terms to be highlighted?
:
: Hi Mark,
:
: It would be great if you can make this change and send the
: QueryTermsExtractor class. I am invoking the QueryScorer(Query)
: contructor. Should I use QueryScorer(Query query, IndexReader reader,
: String fieldName) instead for this to work?
:
: Thanks,
: Harini
:
: mark harwood wrote:
:
: >>>>Is there anyway to restrict the highlighter to
: >>>>
: >>>>
: >>highlight only the values
: >>mentioned for the field 'Content'?
: >>
: >>
: >
: >The problem lies in the QueryTermsExtractor class
: >which is typically used to provide the Highlighter
: >with the list of strings to identify in the text. It
: >currently has no filter for fieldname - you could add
: >this without too much effort.
: >
: >I could make this modification but it may change the
: >behaviour of existing applications - currently the
: >QueryTermsExtractor method that takes a fieldname only
: >uses that fieldname to derive IDF weightings, the
: >proposed change would also have the effect of
: >filtering out any query terms that weren't for this
: >field.
: >Would this change be a problem for anyone?
: >
: >Cheers,
: >Mark
: >
: >--- Harini Raghavan <[EMAIL PROTECTED]>
: >wrote:
: >
: >
: >
: >>Hi,
: >>
: >>I have a requirement to highlight search keywords in
: >>the results and
: >>display the matching fragment of the text with the
: >>results. I am using
: >>the Hits highlighting mentioned in Lucene in Action.
: >>
: >>Here is the search query(BooleanQuery) I am passing
: >>to the IndexSearcher
: >>and QueryScorer:
: >> +DocumentType:news
: >> +(CompanyId:10 CompanyId:20 CompanyId:30
: >>CompanyId:40)
: >> +FilingDate:[20041201 TO 20051201]
: >> +(Content:"cost saving" Content:"cost savings"
: >>Content:outsource
: >>Content:outsources Content:downsize
: >>Content:downsizes
: >>Content:restructuring Content:restructure)
: >>
: >>My requirement is to highlight only the keywords for
: >>'Content' field,
: >>but the highlighter api is also highlighting words
: >>like 'news', '10',
: >>'40' etc.
: >>Is there anyway to restrict the highlighter to
: >>highlight only the values
: >>mentioned for the field 'Content'?
: >>
: >>Thanks,
: >>Harini
: >>
: >>
: >>
: >>
: >>
: >>
: >>
: >>
: >---------------------------------------------------------------------
: >
: >
: >>To unsubscribe, e-mail:
: >>[EMAIL PROTECTED]
: >>For additional commands, e-mail:
: >>[EMAIL PROTECTED]
: >>
: >>
: >>
: >>
: >
: >
: >
: >
: >___________________________________________________________
: >Yahoo! Model Search 2005 - Find the next catwalk superstars -
http://uk.news.yahoo.com/hot/model-search/
: >
: >---------------------------------------------------------------------
: >To unsubscribe, e-mail: [EMAIL PROTECTED]
: >For additional commands, e-mail: [EMAIL PROTECTED]
: >
: >
: >
: >
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: [EMAIL PROTECTED]
: For additional commands, e-mail: [EMAIL PROTECTED]
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
------------------------------------------------------------------------
package org.apache.lucene.search.highlight;
/**
* Copyright 2002-2004 The Apache Software Foundation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import java.io.IOException;
import java.util.Collection;
import java.util.HashSet;
import java.util.Iterator;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.PhraseQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.spans.SpanNearQuery;
/**
* Utility class used to extract the terms used in a query, plus any weights.
* This class will not find terms for MultiTermQuery, RangeQuery and PrefixQuery
classes
* so the caller must pass a rewritten query (see Query.rewrite) to obtain a
list of
* expanded terms.
*
*/
public final class QueryTermExtractor
{
/**
* Extracts all terms texts of a given Query into an array of
WeightedTerms
*
* @param query Query to extract term texts from
* @return an array of the terms used in a query, plus their weights.
*/
public static final WeightedTerm[] getTerms(Query query)
{
return getTerms(query,false,"");
}
/**
* Extracts all terms texts of a given Query into an array of
WeightedTerms
*
* @param query Query to extract term texts from
* @param reader used to compute IDF which can be used to a) score
selected fragments better
* b) use graded highlights eg chaning intensity of font color
* @param fieldName the field on which Inverse Document Frequency (IDF)
calculations are based
* @return an array of the terms used in a query, plus their weights.
*/
public static final WeightedTerm[] getIdfWeightedTerms(Query query,
IndexReader reader, String fieldName)
{
WeightedTerm[] terms=getTerms(query,false,fieldName);
int totalNumDocs=reader.numDocs();
for (int i = 0; i < terms.length; i++)
{
try
{
int docFreq=reader.docFreq(new Term(fieldName,terms[i].term));
//IDF algorithm taken from DefaultSimilarity class
float
idf=(float)(Math.log((float)totalNumDocs/(double)(docFreq+1)) + 1.0);
terms[i].weight*=idf;
}
catch (IOException e)
{
//ignore
}
}
return terms;
}
/**
* Extracts all terms texts of a given Query into an array of
WeightedTerms
*
* @param query Query to extract term texts from
* @param prohibited <code>true</code> to extract "prohibited" terms,
too
* @return an array of the terms used in a query, plus their weights.
*/
public static final WeightedTerm[] getTerms(Query query, boolean
prohibited, String fieldName)
{
HashSet terms=new HashSet();
getTerms(query,terms,prohibited,fieldName);
return (WeightedTerm[]) terms.toArray(new WeightedTerm[0]);
}
private static final void getTerms(Query query, HashSet terms,boolean
prohibited, String fieldName)
{
if (query instanceof BooleanQuery)
getTermsFromBooleanQuery((BooleanQuery) query, terms,
prohibited, fieldName);
else
if (query instanceof PhraseQuery)
getTermsFromPhraseQuery((PhraseQuery) query,
terms, fieldName);
else
if (query instanceof TermQuery)
getTermsFromTermQuery((TermQuery)
query, terms, fieldName);
else
if(query instanceof SpanNearQuery)
getTermsFromSpanNearQuery((SpanNearQuery) query,
terms, fieldName);
}
private static final void getTermsFromBooleanQuery(BooleanQuery query,
HashSet terms, boolean prohibited, String fieldName)
{
BooleanClause[] queryClauses = query.getClauses();
int i;
for (i = 0; i < queryClauses.length; i++)
{
if (prohibited || !queryClauses[i].prohibited)
getTerms(queryClauses[i].query, terms,
prohibited, fieldName);
}
}
private static final void getTermsFromPhraseQuery(PhraseQuery query,
HashSet terms, String fieldName)
{
Term[] queryTerms = query.getTerms();
int i;
String field;
for (i = 0; i < queryTerms.length; i++)
{
if(fieldName.equals(""))
terms.add(new
WeightedTerm(query.getBoost(),queryTerms[i].text()));
else {
field = queryTerms[i].field();
if(field.equals(fieldName))
terms.add(new
WeightedTerm(query.getBoost(),queryTerms[i].text()));
}
}
}
private static final void getTermsFromTermQuery(TermQuery query,
HashSet terms, String fieldName)
{
String field = query.getTerm().field();
if(fieldName.equals(""))
terms.add(new
WeightedTerm(query.getBoost(),query.getTerm().text()));
else if(field.equals(fieldName)) {
terms.add(new
WeightedTerm(query.getBoost(),query.getTerm().text()));
}
}
private static final void getTermsFromSpanNearQuery(SpanNearQuery query,
HashSet terms, String fieldName){
Collection queryTerms = query.getTerms();
for(Iterator iterator = queryTerms.iterator(); iterator.hasNext();){
// break it out for debugging.
Term term = (Term) iterator.next();
String text = term.text();
String field = term.field();
if(fieldName.equals(""))
terms.add(new WeightedTerm(query.getBoost(),
text));
else if(field.equals(fieldName)) {
terms.add(new WeightedTerm(query.getBoost(), text));
}
}
}
}
------------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
