This is great and valuable information, Toke(n)!
Just the other day we recommended this multi-IndexSearcher to
somebody concerned with low QPS rates their benchmarks revealed.
They were hitting their index with a good number of threads and
hitting synchronized blocks in Lucene. Multiple searchers is one
way around that. Also, your sweet spot of 3 makes sense - keeps
all of your cores fully busy.
You are our main SSD info supplier -- keep it coming! :) And let
us know what numbers you get for 2.2 and 2.3, please.
Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
From: Toke Eskildsen <[EMAIL PROTECTED]>
To: java-user@lucene.apache.org
Sent: Thursday, January 17, 2008 5:31:56 AM
Subject: Multiple searchers (Was: CachingWrapperFilter: why cache
per IndexReader?)
On Fri, 2008-01-11 at 11:34 +0100, Toke Eskildsen wrote:
As for shared searcher vs. individual searchers, there was just a
slight penalty for using individual searchers.
Whoops! Seems like I need better QA for my test-code. I didn't use
individual searchers for each thread when I thought I was. The slight
penalty wrongly observed must have been due to measurement variations.
With the corrected test, some interesting observations about our index
can be made, which will definitely affect our configuration. In the
following, the queries/second is an average over 350.000 queries.
For each query, a search is performed and the content of a specific
field is extracted for the first 20 hits.
== System-summary ==
Dual-core Intel Xeon 5148 2.3 GHz, 8 GB RAM, Linux, Lucene 2.1, 37
GB/10
million documents index, queries taken from production system logs.
== Conventional harddisks (2 * 15000 RPM in software RAID 1) ==
1 thread, 1 searcher: 109 queries/sec
2 threads, 1 searcher: 118 queries/sec
2 threads, 2 searchers: 157 queries/sec
3 threads, 1 searcher: 111 queries/sec
3 threads, 3 searchers: 177 queries/sec
4 threads, 1 searcher: 108 queries/sec
4 threads, 4 searchers: 169 queries/sec
== Solid State Drives (2 * 32 GB Samsung in software RAID 0) ==
1 thread, 1 searcher: 193 queries/sec
2 threads, 1 searcher: 295 queries/sec
2 threads, 2 searchers: 357 queries/sec
3 threads, 1 searcher: 197 queries/sec
3 threads, 3 searchers: 369 queries/sec
4 threads, 1 searcher: 192 queries/sec
4 threads, 4 searchers: 302 queries/sec
Graphs can be viewed at http://wiki.statsbiblioteket.dk/summa/Hardware
For our setup it seems that the usual avoid-multiple-searchers advice
is
not valid, neither for conventional harddisks, nor Solid State Drives.
The optimal configuration for our dual-core test machine is three
threads with individual searchers. The obvious question is whether
this
can be extended to other cases.
As for threading, I noticed something strange: On the dual-core
machine, two threads gave better performance than one, while 4
threads
gave the same performance as one.
As can be seen above, this strange picture is consistent. 1, 3 and 4
threads with shared searcher performs the same, independent of which
storage the machine uses, while 2 threads performs markedly better.
I've started the same test-suite for Lucene 2.2 and 2.3RC2. It should
be finished in a day or two.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
