Thanks for the help Artem and Jokin, I let my index "warm up" and found exactly the results I was looking for! The improvement was fairly minor at first:
Found 500 documents in 11 seconds, 46 requests/second Found 500 documents in 10 seconds, 48 requests/second Found 500 documents in 10 seconds, 50 requests/second Found 500 documents in 9 seconds, 53 requests/second ... Found 500 documents in 9 seconds, 58 requests/second Found 500 documents in 8 seconds, 62 requests/second When I increased the number of documents for each round I was quite impressed: Found 10000 documents in 36 seconds, 281 requests/second Found 10000 documents in 4 seconds, 2,498 requests/second Found 10000 documents in 3 seconds, 3,991 requests/second Found 10000 documents in 2 seconds, 4,878 requests/second Found 10000 documents in 2 seconds, 6,271 requests/second Found 10000 documents in 1 seconds, 7,066 requests/second Found 10000 documents in 1 seconds, 7,759 requests/second Found 10000 documents in 1 seconds, 7,685 requests/second Found 10000 documents in 1 seconds, 8,105 requests/second Found 10000 documents in 1 seconds, 8,289 requests/second Found 10000 documents in 1 seconds, 8,355 requests/second That's with just a single thread accessing IndexSearcher. I think this experiment also answers a lot of my other questions to the list over the past few days with regards to QueryParser being "slow"...I just need to let my index warm up a bit longer. Is it possible to serialize the internal caches of an IndexSearcher/IndexReader so when its re-opened it doesn't require the warm up period? Jokin, I agree this test wasn't very representative of a normal search. I wanted to set a baseline. I was concerned that if the fastest possible search was 40 requests/second then adding QueryParsers, geocoding, spellchecking, etc. would require me to look elsewhere for a solution. As a new user to Lucene, I'm finding the Solr Wiki to be quite helpful explaining the role of tokens, analyzers, filters, etc. Thanks, Ron ----- Original Message ---- From: Artem Chereisky <a.cherei...@gmail.com> To: lucene-net-user@incubator.apache.org Sent: Tue, November 3, 2009 4:35:12 PM Subject: Re: 26m documents, 8 fields, Core2 Duo (E6550 2.33GHz), /tags/Lucene.Net_2_4_0, 40 requests/second...is that fast or slow? My index has 2M documents. Any search with a term query and a simple custom hit collector takes under 20ms. How large is your index? Can you use RAMDirectory and make IndexReader read-only. That speeds things up. On Tue, Nov 3, 2009 at 10:05 PM, Jokin Cuadrado <joki...@gmail.com> wrote: > Lucene has a "cold start time" depending how big is your index, I > think that you should make some searches (50 for example) for warming > all the internal caches. > > As you are searching just for an exact match that doesn't return much > results, the throughput should be higher. > > However, the test that you are making is not very representative, > because you are using Lucene like a giant hashtable, without knowing > what are going to be the usual queries and the average hit number, > it's difficult to guess any optimization suggestion. > > On Tue, Nov 3, 2009 at 4:58 AM, Ron Grabowski <rongrabow...@yahoo.com> > wrote: > > searcher > > -- > Jokin >
