I am not sure why you are getting all 60k docs at a time. If you use
the Hits object, it caches the top 50 or so, but doesn't retrieve all
the documents at once.
Also, what are the size of your fields and how many fields do you have
per document?
Have you done any profiling to find the bottlenecks? An index size of
50mb is actually pretty small for Lucene, perhaps you can share more
about your setup.
-Grant
Satuluri, Venu_Madhav wrote:
Hi,
I am looking for ways to improve the performance of lucene search in our
app. Lucene performance is visibly slow when there are a lot of
documents to be returned (performance almost seems directly proportional
to the number of documents returned by Searcher). However, we show 20
results per page, so it seems to be a waste of time to get all, say,
60,000 documents when all I need are the nth 20 (i.e. if user requests
4th page of results, I just need documents 61 to 80). I've tried using
the Searcher.search() method that returns TopFieldDocs. This method
works much faster than the ordinary Searcher.search() that returns all
the results. The trouble with this method is that I cant use it to get
an arbitrary portion of the results, I can only get the top few docs.
Our index size is around 50 MB. Its optimized every one hour. I have
tried a RAMDirectory, but even though using this improves performance by
upto 2 times, its not good enough. Also our index gets modified by
multiple processes so keeping the RAMDirectory up-to-date is a hassle.
Thanks,
Venu
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Grant Ingersoll
Sr. Software Engineer
Center for Natural Language Processing
Syracuse University
School of Information Studies
335 Hinds Hall
Syracuse, NY 13244
http://www.cnlp.org
Voice: 315-443-5484
Fax: 315-443-6886
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
