Great news for Solr -- a third party library that I'm calling is serialized.
Silly me, I made a mistake when ruling out that library as the culprit
earlier. Solr itself scales just great as add threads. JProfiler helped me
find the problem.
Sorry for the false alarm, and thanks for the
Thank you Grant and Lance for your comments -- I've run into a separate snag
which puts this on hold for a bit, but I'll return to finish digging into
this and post my results. - Michael
On Thu, Sep 24, 2009 at 9:23 PM, Lance Norskog goks...@gmail.com wrote:
Are you on Java 5, 6 or 7? Each
Are you on Java 5, 6 or 7? Each release sees some tweaking of the Java
multithreading model as well as performance improvements (and bug
fixes) in the Sun HotSpot runtime.
You may be tripping over the TCP/IP multithreaded connection manager.
You might wish to create each client thread with a
For 8-CPU load-stress testing of Tomcat you are probably making mistake:
- you should execute load-stress software and wait 5-30 minutes (depends on
index size) BEFORE taking measurements.
1. JVM HotSpot need to compile everything into native code
2. Tomcat Thread Pool needs warm up
3. SOLR
I'm using a Solr 1.4 nightly from around July. Is that recent enough to
have the improved reader implementation?
I'm not sure whether you'd call my operations IO heavy -- each query has so
many terms (~50) that even against a 45K document index a query takes 130ms,
but the entire index is in a
]
Sent: September-23-09 11:17 AM
To: solr-user@lucene.apache.org; yo...@lucidimagination.com
Subject: Re: Parallel requests to Tomcat
I'm using a Solr 1.4 nightly from around July. Is that recent enough to
have the improved reader implementation?
I'm not sure whether you'd call my operations IO
Hi Fuad, thanks for the reply.
My queries are heavy enough that the difference in performance is obvious.
I am using a home-grown load testing script that sends 1000 realistic
queries to the server and takes the average response time. My index is on a
ramfs which I've shown makes the QR and doc
[mailto:solrco...@gmail.com]
Sent: September-23-09 11:17 AM
To: solr-user@lucene.apache.org; yo...@lucidimagination.com
Subject: Re: Parallel requests to Tomcat
I'm using a Solr 1.4 nightly from around July. Is that recent enough to
have the improved reader implementation?
I'm not sure whether
On Wed, Sep 23, 2009 at 11:26 AM, Fuad Efendi f...@efendi.ca wrote:
- something obviously wrong in your case, 130ms is too high. Is it
dedicated
server? Disk swapping? Etc.
It's that my queries are ridiculously complex. My users are very familiar
with boolean searching, and I'm doing a lot
8 queries against 1 Tomcat average 600ms per query, while 8 queries
against
8 Tomcats average 190ms per query (on a dedicated 8 CPU server w 32G RAM).
I don't see how to interpret these numbers except that Tomcat is not
multithreading as well as it should :)
Hi Michael, I think it is very
On Wed, Sep 23, 2009 at 11:17 AM, Michael solrco...@gmail.com wrote:
I'm using a Solr 1.4 nightly from around July. Is that recent enough to
have the improved reader implementation?
I'm not sure whether you'd call my operations IO heavy -- each query has so
many terms (~50) that even against
Hi Fuad,
On Wed, Sep 23, 2009 at 11:37 AM, Fuad Efendi f...@efendi.ca wrote:
8 queries against 1 Tomcat average 600ms per query, while 8 queries
against
8 Tomcats average 190ms per query (on a dedicated 8 CPU server w 32G
RAM).
I don't see how to interpret these numbers except that
Hi Yonik,
On Wed, Sep 23, 2009 at 11:42 AM, Yonik Seeley
yo...@lucidimagination.comwrote:
This could well be IO bound - lots of seeks and reads.
If this were IO bound, wouldn't I see the same results when sending my 8
requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm
This sure seems like a good time to try LucidGaze for Solr. That would
give some Solr-specific profiling data.
http://www.lucidimagination.com/Downloads/LucidGaze-for-Solr
wunder
On Sep 23, 2009, at 8:47 AM, Michael wrote:
Hi Yonik,
On Wed, Sep 23, 2009 at 11:42 AM, Yonik Seeley
Thanks for the suggestion, Walter! I've been using Gaze 1.0 for a while
now, but when I moved to a multicore approach (which was the impetus behind
all of this testing) Gaze failed to start and I had to comment it out of
solrconfig.xml to get Solr to start. Are you aware whether Gaze is able to
On Wed, Sep 23, 2009 at 11:47 AM, Michael solrco...@gmail.com wrote:
Hi Yonik,
On Wed, Sep 23, 2009 at 11:42 AM, Yonik Seeley yo...@lucidimagination.com
wrote:
This could well be IO bound - lots of seeks and reads.
If this were IO bound, wouldn't I see the same results when sending my 8
On Wed, Sep 23, 2009 at 12:05 PM, Yonik Seeley
yo...@lucidimagination.comwrote:
On Wed, Sep 23, 2009 at 11:47 AM, Michael solrco...@gmail.com wrote:
If this were IO bound, wouldn't I see the same results when sending my 8
requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm
8 threads sharing something may have *some* overhead versus 8 processes,
but
as you say, 410ms overhead points to a different problem.
- You have baseline (single-threaded load-stress script sending requests to
SOLR) (1-request-in-parallel, 8 requests to 8 Tomcats); 200ms looks
extremely
I'm not sure whether you'd call my operations IO heavy -- each query has
so
many terms (~50) that even against a 45K document index a query takes
130ms,
but the entire index is in a ramfs.
The more terms, the more it takes to find docset intersections (belonging to
each term); something in
What version of Solr are you using?
Solr1.3 and Lucene 2.4 defaulted to an index reader implementation
that had to synchronize, so search operations that are IO heavy
can't proceed in parallel. You shouldn't see this with 1.4
-Yonik
http://www.lucidimagination.com
On Tue, Sep 22, 2009 at 4:03
20 matches
Mail list logo
