(10/05/05 22:08), Serdar Sahin wrote:
Hi,
Currently, there are similar topics active in the mailing list, but it I did
not want to steal the topic.
I have currently indexed 100.000 documents, they are microsoft office/pdf
etc documents I convert them to TXT files before indexing. Files are between
1-500 pages. When I search something and filter it to retrieve documents
that has more than 100 pages, and activate highlighting, it takes 0.8-3
seconds, depending on the query. (10 result per page) If I retrieve
documents that has 1-5 pages, it drops to 0.1 seconds.
If I disable highlighting, it drops to 0.1-0.2 seconds, even on the large
documents, which is more than enough. This problem mostly happens where
there are no caches, on the first query. I use this configuration for
highlighting:
$query->addHighlightField('description')->addHighlightField('plainText');
$query->setHighlightSimplePre('<strong>');
$query->setHighlightSimplePost('</strong>');
$query->setHighlightHighlightMultiTerm(TRUE);
$query->setHighlightMaxAnalyzedChars(10000);
$query->setHighlightSnippets(2);
Do you have any suggestions to improve response time while highlighting is
active? I have read couple of articles you have previously provided but they
did not help.
And for the second question, I retrieve these fields:
$query->addField('title')->addField('cat')->addField('thumbs_up')->
addField('thumbs_down')->addField('lang')->addField('id')->
addField('username')->addField('view_count')->addField('pages')->
addField('no_img')->addField('date');
If I can't solve the highlighting problem on large documents, I can simply
disable it and retrieve first x characters from the plainText (full text)
field, but is it possible to retrieve first x characters without using the
highlighting feature? When I use this;
$query->setHighlight(TRUE);
$query->setHighlightAlternateField('plainText');
$query->setHighlightMaxAnalyzedChars(0);
$query->setHighlightMaxAlternateFieldLength(256);
It still takes 2 seconds if I retrieve 10 rows that has 200-300 pages. The
highlighting still works so it might be the source of the problem, I want to
completely disable it and retrieve only the first 256 characters of the
plainText field. Is it possible? It may remove some overhead give better
performance.
I personally prefer the highlighting solution but I also would like to hear
the solution for this problem. For the same query, if I disable highlighting
and without retrieving (but still searching) the plainText field, it drops
to 0.0094 seconds. So I think if I can get the first 256 characters without
using the highlighting, I will get better performance.
Any suggestions regarding with these two problems will highly appreciated.
Thanks,
Serdar Sahin
Hi Serdar,
There are a few things I think of you can try.
1. Provide another field for highlighting and use copyField
to copy plainText to the highlighting field. When using copyField,
specify maxChars attribute to limit the length of the copy of plainText.
This should work on Solr 1.4.
2. If you can use branch_3x version of Solr, try FastVectorHighlighter.
Koji
--
http://www.rondhuit.com/en/