Re: slow highlighting because of stemming

[Orosz György]

Thanks for the answers!
This was the solution! :) (my fault was that I tried to use the "on" value
instead of true - don't know why..)
Gyuri

2011/7/30 Michael Sokolov 

> On 7/30/2011 3:46 AM, Orosz György wrote:
>
>> Hi,
>>
>> Thanks for the answer!
>> I am doing some logging about stemming, and what I can see is that a lot
>> of
>> tokens are stemmed for the highlighting. It is the strange part, since I
>> don't understand why does any highlighter need stemming again.
>>
> Consider that the highlighter needs to match terms from the query with
> terms from the document, just like search. If the indexed document has been
> stemmed, then the query also needs to be stemmed, or you won't see matches.
>
> -Mike
>

Re: slow highlighting because of stemming

[Michael Sokolov]

On 7/30/2011 3:46 AM, Orosz György wrote:

Hi,

Thanks for the answer!
I am doing some logging about stemming, and what I can see is that a lot of
tokens are stemmed for the highlighting. It is the strange part, since I
don't understand why does any highlighter need stemming again.
Consider that the highlighter needs to match terms from the query with 
terms from the document, just like search. If the indexed document has 
been stemmed, then the query also needs to be stemmed, or you won't see 
matches.


-Mike

Re: slow highlighting because of stemming

[Ahmet Arslan]

> I am doing some logging about stemming, and what I can see
> is that a lot of
> tokens are stemmed for the highlighting. It is the strange
> part, since I
> don't understand why does any highlighter need stemming
> again.

Highlighting do re-analyze the text being highlighted.

> Anyway my docments are not really large, just a few
> kilobytes, but thanks
> for this suggestion.
> 
> If you could help me in "how could I just ignore the
> stemming for
> highlighting" thing it would be very great!

If you store term vectors, the this re-analyze is skipped.
http://wiki.apache.org/solr/FieldOptionsByUseCase

Re: slow highlighting because of stemming

[Orosz György]

Hi,

Thanks for the answer!
I am doing some logging about stemming, and what I can see is that a lot of
tokens are stemmed for the highlighting. It is the strange part, since I
don't understand why does any highlighter need stemming again.
Anyway my docments are not really large, just a few kilobytes, but thanks
for this suggestion.

If you could help me in "how could I just ignore the stemming for
highlighting" thing it would be very great!

Thanks,
Gyuri

2011/7/29 Mike Sokolov 

> I'm not sure I would identify stemming as the culprit here.
>
> Do you have very large documents?  If so, there is a patch for FVH
> committed to limit the number of phrases it looks at; see hl.phraseLimit,
> but this won't be available until 3.4 is released.


> You can also limit the amount of each document that is analyzed by the
> regular Highlighter using maxDocCharsToAnalyze (and maybe this applies to
> FVH? not sure)
>
> Using RegexFragmenter is also probably slower than something like
> SimpleFragmenter.
>
> There is work to implement faster highlighting for Solr/Lucene, but it
> depends on some basic changes to the search architecture so it might be a
> while before that becomes available.  See https://issues.apache.org/**
> jira/browse/LUCENE-3318if 
> you're interested in following that development.
>
> -Mike
>
>
> On 07/29/2011 04:55 AM, Orosz György wrote:
>
>> Dear all,
>>
>> I am quite new about using Solr, but would like to ask your help.
>> I am developing an application which should be able to highlight the
>> results
>> of a query. For this I am using regex fragmenter:
>> 
>>> class="org.apache.solr.**highlight.RegexFragmenter">
>> 
>>   500
>>   0.5
>>   <**/str>
>>  

Re: slow highlighting because of stemming

[Mike Sokolov]

I'm not sure I would identify stemming as the culprit here.

Do you have very large documents?  If so, there is a patch for FVH 
committed to limit the number of phrases it looks at; see 
hl.phraseLimit, but this won't be available until 3.4 is released.


You can also limit the amount of each document that is analyzed by the 
regular Highlighter using maxDocCharsToAnalyze (and maybe this applies 
to FVH? not sure)


Using RegexFragmenter is also probably slower than something like 
SimpleFragmenter.


There is work to implement faster highlighting for Solr/Lucene, but it 
depends on some basic changes to the search architecture so it might be 
a while before that becomes available.  See 
https://issues.apache.org/jira/browse/LUCENE-3318 if you're interested 
in following that development.


-Mike

On 07/29/2011 04:55 AM, Orosz György wrote:

Dear all,

I am quite new about using Solr, but would like to ask your help.
I am developing an application which should be able to highlight the results
of a query. For this I am using regex fragmenter:


 
   500
   0.5
   
  
  true
   [-\w ,/\n\"']{20,300}[.?!]
   dokumentum_syn_query
 

   
The field is indexed with term vectors and offsets:

 
   
 
 
  
 
   
   
 
  
  
 
 
   
 

The highlighting works well, excepts that its really slow. I realized that
this is because the highlighter/fragmenter does stemming for all the results
documents again.

Could you please help me why does it happen an how should I avoid this? (I
thought that using fastvectorhighlighter will solve my problem, but it
didn't)

Thanks in advance!
Gyuri Orosz

   

slow highlighting because of stemming

[Orosz György]

Dear all,

I am quite new about using Solr, but would like to ask your help.
I am developing an application which should be able to highlight the results
of a query. For this I am using regex fragmenter:

   

  500
  0.5
  
 
 true
  [-\w ,/\n\"']{20,300}[.?!]
  dokumentum_syn_query

   
  
The field is indexed with term vectors and offsets:


  


 

  
  

 
 


  


The highlighting works well, excepts that its really slow. I realized that
this is because the highlighter/fragmenter does stemming for all the results
documents again.

Could you please help me why does it happen an how should I avoid this? (I
thought that using fastvectorhighlighter will solve my problem, but it
didn't)

Thanks in advance!
Gyuri Orosz