Re: Parallel requests to Tomcat
Great news for Solr -- a third party library that I'm calling is serialized. Silly me, I made a mistake when ruling out that library as the culprit earlier. Solr itself scales just great as add threads. JProfiler helped me find the problem. Sorry for the false alarm, and thanks for the suggestions! Michael On Fri, Sep 25, 2009 at 10:02 AM, Michael solrco...@gmail.com wrote: Thank you Grant and Lance for your comments -- I've run into a separate snag which puts this on hold for a bit, but I'll return to finish digging into this and post my results. - Michael On Thu, Sep 24, 2009 at 9:23 PM, Lance Norskog goks...@gmail.com wrote: Are you on Java 5, 6 or 7? Each release sees some tweaking of the Java multithreading model as well as performance improvements (and bug fixes) in the Sun HotSpot runtime. You may be tripping over the TCP/IP multithreaded connection manager. You might wish to create each client thread with a separate socket. Also, here is a standard bit of benchmarking advice: include think time. This means that instead of sending requests constantly, each thread should time out for a few seconds before sending the next request. This simulates a user stopping and thinking before clicking the mouse again. This helps simulate the quantity of threads, etc. which are stopped and waiting at each stage of the request pipeline. As it is, you are trying to simulate the throughput behaviour without simulating the horizontal volume. (Benchmarking is much harder than it looks.) On Wed, Sep 23, 2009 at 9:43 AM, Grant Ingersoll gsing...@apache.org wrote: On Sep 23, 2009, at 12:09 PM, Michael wrote: On Wed, Sep 23, 2009 at 12:05 PM, Yonik Seeley yo...@lucidimagination.comwrote: On Wed, Sep 23, 2009 at 11:47 AM, Michael solrco...@gmail.com wrote: If this were IO bound, wouldn't I see the same results when sending my 8 requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm querying 8 processes or 8 threads in 1 process, right? Right - I was thinking IO bound at the Lucene Directory level - which synchronized in the past and led to poor concurrency. Buy your Solr version is recent enough to use the newer unsynchronized method by default (on non-windows) Ah, OK. So it looks like comparing to Jetty is my only next step. Although I'm not sure what I'm going to do based on the result of that test -- if Jetty behaves differently, then I still don't know why the heck Tomcat is behaving badly! :) Have you done any profiling to see where hotspots are? Have you looked at garbage collection? Do you have any full collections occurring? What garbage collector are you using? How often are you updating/committing, etc? -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search -- Lance Norskog goks...@gmail.com
Re: Parallel requests to Tomcat
Thank you Grant and Lance for your comments -- I've run into a separate snag which puts this on hold for a bit, but I'll return to finish digging into this and post my results. - Michael On Thu, Sep 24, 2009 at 9:23 PM, Lance Norskog goks...@gmail.com wrote: Are you on Java 5, 6 or 7? Each release sees some tweaking of the Java multithreading model as well as performance improvements (and bug fixes) in the Sun HotSpot runtime. You may be tripping over the TCP/IP multithreaded connection manager. You might wish to create each client thread with a separate socket. Also, here is a standard bit of benchmarking advice: include think time. This means that instead of sending requests constantly, each thread should time out for a few seconds before sending the next request. This simulates a user stopping and thinking before clicking the mouse again. This helps simulate the quantity of threads, etc. which are stopped and waiting at each stage of the request pipeline. As it is, you are trying to simulate the throughput behaviour without simulating the horizontal volume. (Benchmarking is much harder than it looks.) On Wed, Sep 23, 2009 at 9:43 AM, Grant Ingersoll gsing...@apache.org wrote: On Sep 23, 2009, at 12:09 PM, Michael wrote: On Wed, Sep 23, 2009 at 12:05 PM, Yonik Seeley yo...@lucidimagination.comwrote: On Wed, Sep 23, 2009 at 11:47 AM, Michael solrco...@gmail.com wrote: If this were IO bound, wouldn't I see the same results when sending my 8 requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm querying 8 processes or 8 threads in 1 process, right? Right - I was thinking IO bound at the Lucene Directory level - which synchronized in the past and led to poor concurrency. Buy your Solr version is recent enough to use the newer unsynchronized method by default (on non-windows) Ah, OK. So it looks like comparing to Jetty is my only next step. Although I'm not sure what I'm going to do based on the result of that test -- if Jetty behaves differently, then I still don't know why the heck Tomcat is behaving badly! :) Have you done any profiling to see where hotspots are? Have you looked at garbage collection? Do you have any full collections occurring? What garbage collector are you using? How often are you updating/committing, etc? -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search -- Lance Norskog goks...@gmail.com
Re: Parallel requests to Tomcat
Are you on Java 5, 6 or 7? Each release sees some tweaking of the Java multithreading model as well as performance improvements (and bug fixes) in the Sun HotSpot runtime. You may be tripping over the TCP/IP multithreaded connection manager. You might wish to create each client thread with a separate socket. Also, here is a standard bit of benchmarking advice: include think time. This means that instead of sending requests constantly, each thread should time out for a few seconds before sending the next request. This simulates a user stopping and thinking before clicking the mouse again. This helps simulate the quantity of threads, etc. which are stopped and waiting at each stage of the request pipeline. As it is, you are trying to simulate the throughput behaviour without simulating the horizontal volume. (Benchmarking is much harder than it looks.) On Wed, Sep 23, 2009 at 9:43 AM, Grant Ingersoll gsing...@apache.org wrote: On Sep 23, 2009, at 12:09 PM, Michael wrote: On Wed, Sep 23, 2009 at 12:05 PM, Yonik Seeley yo...@lucidimagination.comwrote: On Wed, Sep 23, 2009 at 11:47 AM, Michael solrco...@gmail.com wrote: If this were IO bound, wouldn't I see the same results when sending my 8 requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm querying 8 processes or 8 threads in 1 process, right? Right - I was thinking IO bound at the Lucene Directory level - which synchronized in the past and led to poor concurrency. Buy your Solr version is recent enough to use the newer unsynchronized method by default (on non-windows) Ah, OK. So it looks like comparing to Jetty is my only next step. Although I'm not sure what I'm going to do based on the result of that test -- if Jetty behaves differently, then I still don't know why the heck Tomcat is behaving badly! :) Have you done any profiling to see where hotspots are? Have you looked at garbage collection? Do you have any full collections occurring? What garbage collector are you using? How often are you updating/committing, etc? -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search -- Lance Norskog goks...@gmail.com
RE: Parallel requests to Tomcat
For 8-CPU load-stress testing of Tomcat you are probably making mistake: - you should execute load-stress software and wait 5-30 minutes (depends on index size) BEFORE taking measurements. 1. JVM HotSpot need to compile everything into native code 2. Tomcat Thread Pool needs warm up 3. SOLR caches need warm up(!) And etc. 8 parallel requests are too small for default Tomcat; it uses 150 threads (default for old versions), and new Concurrent package from Java 5 You should not test manually; use software such as The Grinder etc., also note please: there is difference between mean time and response time, between average (successful) requests per second and average response time... Tomcat is serializing the requests - doesn't mean anything for performance... yes, it has dedicated Listener on dedicated port dispatching requests to worker threads... and LAN NIC card serializes everything too... Fuad Efendi http://www.linkedin.com/in/liferay -Original Message- From: Michael [mailto:solrco...@gmail.com] Sent: September-22-09 4:04 PM To: solr-user@lucene.apache.org Subject: Parallel requests to Tomcat Hi, I have a Solr+Tomcat installation on an 8 CPU Linux box, and I just tried sending parallel requests to it and measuring response time. I would expect that it could handle up to 8 parallel requests without significant slowdown of any individual request. Instead, I found that Tomcat is serializing the requests. For example, the response time for each of 2 parallel requests is nearly 2 times that for a single request, and the time for each of 8 parallel requests is about 4 times that of a single request. I am pretty sure this is a Tomcat issue, for when I started 8 identical instances of Solr+Tomcat on the machine (on 8 different ports), I could send one request to each in parallel with only a 20% slowdown (compared to 300% in a single Tomcat.) I'm using the stock Tomcat download with minimal configuration changes, except that I disabled all logging (in case the logger was blocking for each request, serializing them.) I'm giving 2G RAM to each JVM. Does anyone more familiar with Tomcat know what's wrong? I can't imagine that Tomcat really can't handle parallel requests.
Re: Parallel requests to Tomcat
I'm using a Solr 1.4 nightly from around July. Is that recent enough to have the improved reader implementation? I'm not sure whether you'd call my operations IO heavy -- each query has so many terms (~50) that even against a 45K document index a query takes 130ms, but the entire index is in a ramfs. - Michael On Tue, Sep 22, 2009 at 8:08 PM, Yonik Seeley yo...@lucidimagination.comwrote: What version of Solr are you using? Solr1.3 and Lucene 2.4 defaulted to an index reader implementation that had to synchronize, so search operations that are IO heavy can't proceed in parallel. You shouldn't see this with 1.4 -Yonik http://www.lucidimagination.com On Tue, Sep 22, 2009 at 4:03 PM, Michael solrco...@gmail.com wrote: Hi, I have a Solr+Tomcat installation on an 8 CPU Linux box, and I just tried sending parallel requests to it and measuring response time. I would expect that it could handle up to 8 parallel requests without significant slowdown of any individual request. Instead, I found that Tomcat is serializing the requests. For example, the response time for each of 2 parallel requests is nearly 2 times that for a single request, and the time for each of 8 parallel requests is about 4 times that of a single request. I am pretty sure this is a Tomcat issue, for when I started 8 identical instances of Solr+Tomcat on the machine (on 8 different ports), I could send one request to each in parallel with only a 20% slowdown (compared to 300% in a single Tomcat.) I'm using the stock Tomcat download with minimal configuration changes, except that I disabled all logging (in case the logger was blocking for each request, serializing them.) I'm giving 2G RAM to each JVM. Does anyone more familiar with Tomcat know what's wrong? I can't imagine that Tomcat really can't handle parallel requests.
RE: Parallel requests to Tomcat
I have 0-15ms for 50M (millions docs), Tomcat, 8-CPU: http://www.tokenizer.org == - something obviously wrong in your case, 130ms is too high. Is it dedicated server? Disk swapping? Etc. -Original Message- From: Michael [mailto:solrco...@gmail.com] Sent: September-23-09 11:17 AM To: solr-user@lucene.apache.org; yo...@lucidimagination.com Subject: Re: Parallel requests to Tomcat I'm using a Solr 1.4 nightly from around July. Is that recent enough to have the improved reader implementation? I'm not sure whether you'd call my operations IO heavy -- each query has so many terms (~50) that even against a 45K document index a query takes 130ms, but the entire index is in a ramfs. - Michael On Tue, Sep 22, 2009 at 8:08 PM, Yonik Seeley yo...@lucidimagination.comwrote: What version of Solr are you using? Solr1.3 and Lucene 2.4 defaulted to an index reader implementation that had to synchronize, so search operations that are IO heavy can't proceed in parallel. You shouldn't see this with 1.4 -Yonik http://www.lucidimagination.com On Tue, Sep 22, 2009 at 4:03 PM, Michael solrco...@gmail.com wrote: Hi, I have a Solr+Tomcat installation on an 8 CPU Linux box, and I just tried sending parallel requests to it and measuring response time. I would expect that it could handle up to 8 parallel requests without significant slowdown of any individual request. Instead, I found that Tomcat is serializing the requests. For example, the response time for each of 2 parallel requests is nearly 2 times that for a single request, and the time for each of 8 parallel requests is about 4 times that of a single request. I am pretty sure this is a Tomcat issue, for when I started 8 identical instances of Solr+Tomcat on the machine (on 8 different ports), I could send one request to each in parallel with only a 20% slowdown (compared to 300% in a single Tomcat.) I'm using the stock Tomcat download with minimal configuration changes, except that I disabled all logging (in case the logger was blocking for each request, serializing them.) I'm giving 2G RAM to each JVM. Does anyone more familiar with Tomcat know what's wrong? I can't imagine that Tomcat really can't handle parallel requests.
Re: Parallel requests to Tomcat
Hi Fuad, thanks for the reply. My queries are heavy enough that the difference in performance is obvious. I am using a home-grown load testing script that sends 1000 realistic queries to the server and takes the average response time. My index is on a ramfs which I've shown makes the QR and doc caches unnecessary; I am warming up the filter and fieldvalue caches before beginning the test. There's no appreciable difference between query times at the beginning, middle, or end of the test, so I can't blame the hotspot or the Tomcat thread pool for not being warmed up. The queries I'm using are complex enough that they take a long time to run. 8 queries against 1 Tomcat average 600ms per query, while 8 queries against 8 Tomcats average 190ms per query (on a dedicated 8 CPU server w 32G RAM). I don't see how to interpret these numbers except that Tomcat is not multithreading as well as it should :) Your thoughts? Michael On Wed, Sep 23, 2009 at 10:48 AM, Fuad Efendi f...@efendi.ca wrote: For 8-CPU load-stress testing of Tomcat you are probably making mistake: - you should execute load-stress software and wait 5-30 minutes (depends on index size) BEFORE taking measurements. 1. JVM HotSpot need to compile everything into native code 2. Tomcat Thread Pool needs warm up 3. SOLR caches need warm up(!) And etc. 8 parallel requests are too small for default Tomcat; it uses 150 threads (default for old versions), and new Concurrent package from Java 5 You should not test manually; use software such as The Grinder etc., also note please: there is difference between mean time and response time, between average (successful) requests per second and average response time... Tomcat is serializing the requests - doesn't mean anything for performance... yes, it has dedicated Listener on dedicated port dispatching requests to worker threads... and LAN NIC card serializes everything too... Fuad Efendi http://www.linkedin.com/in/liferay -Original Message- From: Michael [mailto:solrco...@gmail.com] Sent: September-22-09 4:04 PM To: solr-user@lucene.apache.org Subject: Parallel requests to Tomcat Hi, I have a Solr+Tomcat installation on an 8 CPU Linux box, and I just tried sending parallel requests to it and measuring response time. I would expect that it could handle up to 8 parallel requests without significant slowdown of any individual request. Instead, I found that Tomcat is serializing the requests. For example, the response time for each of 2 parallel requests is nearly 2 times that for a single request, and the time for each of 8 parallel requests is about 4 times that of a single request. I am pretty sure this is a Tomcat issue, for when I started 8 identical instances of Solr+Tomcat on the machine (on 8 different ports), I could send one request to each in parallel with only a 20% slowdown (compared to 300% in a single Tomcat.) I'm using the stock Tomcat download with minimal configuration changes, except that I disabled all logging (in case the logger was blocking for each request, serializing them.) I'm giving 2G RAM to each JVM. Does anyone more familiar with Tomcat know what's wrong? I can't imagine that Tomcat really can't handle parallel requests.
RE: Parallel requests to Tomcat
Correction: 0 - 150ms (depends on size of query results; 150ms for non-cached (new) queries returning more than 50K docs). -Original Message- From: Fuad Efendi [mailto:f...@efendi.ca] Sent: September-23-09 11:26 AM To: solr-user@lucene.apache.org Subject: RE: Parallel requests to Tomcat I have 0-15ms for 50M (millions docs), Tomcat, 8-CPU: http://www.tokenizer.org == - something obviously wrong in your case, 130ms is too high. Is it dedicated server? Disk swapping? Etc. -Original Message- From: Michael [mailto:solrco...@gmail.com] Sent: September-23-09 11:17 AM To: solr-user@lucene.apache.org; yo...@lucidimagination.com Subject: Re: Parallel requests to Tomcat I'm using a Solr 1.4 nightly from around July. Is that recent enough to have the improved reader implementation? I'm not sure whether you'd call my operations IO heavy -- each query has so many terms (~50) that even against a 45K document index a query takes 130ms, but the entire index is in a ramfs. - Michael On Tue, Sep 22, 2009 at 8:08 PM, Yonik Seeley yo...@lucidimagination.comwrote: What version of Solr are you using? Solr1.3 and Lucene 2.4 defaulted to an index reader implementation that had to synchronize, so search operations that are IO heavy can't proceed in parallel. You shouldn't see this with 1.4 -Yonik http://www.lucidimagination.com On Tue, Sep 22, 2009 at 4:03 PM, Michael solrco...@gmail.com wrote: Hi, I have a Solr+Tomcat installation on an 8 CPU Linux box, and I just tried sending parallel requests to it and measuring response time. I would expect that it could handle up to 8 parallel requests without significant slowdown of any individual request. Instead, I found that Tomcat is serializing the requests. For example, the response time for each of 2 parallel requests is nearly 2 times that for a single request, and the time for each of 8 parallel requests is about 4 times that of a single request. I am pretty sure this is a Tomcat issue, for when I started 8 identical instances of Solr+Tomcat on the machine (on 8 different ports), I could send one request to each in parallel with only a 20% slowdown (compared to 300% in a single Tomcat.) I'm using the stock Tomcat download with minimal configuration changes, except that I disabled all logging (in case the logger was blocking for each request, serializing them.) I'm giving 2G RAM to each JVM. Does anyone more familiar with Tomcat know what's wrong? I can't imagine that Tomcat really can't handle parallel requests.
Re: Parallel requests to Tomcat
On Wed, Sep 23, 2009 at 11:26 AM, Fuad Efendi f...@efendi.ca wrote: - something obviously wrong in your case, 130ms is too high. Is it dedicated server? Disk swapping? Etc. It's that my queries are ridiculously complex. My users are very familiar with boolean searching, and I'm doing a lot of processing outside of Solr that increases the query size by something like 50x. I'm OK with the individual query time -- I can always shave terms off if I must. It's the difference between 1 Tomcat and 8 Tomcats that is the problem: I'd like to be able to harness all 8 CPUs! While my test corpus is 45K docs, my actual corpus will be 30MM, and so I'd like to get all the performance I can out of my box. Michael
RE: Parallel requests to Tomcat
8 queries against 1 Tomcat average 600ms per query, while 8 queries against 8 Tomcats average 190ms per query (on a dedicated 8 CPU server w 32G RAM). I don't see how to interpret these numbers except that Tomcat is not multithreading as well as it should :) Hi Michael, I think it is very natural; 8 single processes not sharing anything are faster than 8 threads sharing something. However, 600ms is too high. My index is on a ramfs which I've shown makes the QR and doc caches unnecessary; However, SOLR is faster than pure Lucene, try SOLR caches! (I am not sure about current version of SOLR(Lucene), but Lucene always used synchronized isDeleted() method which causes 'serialization').
Re: Parallel requests to Tomcat
On Wed, Sep 23, 2009 at 11:17 AM, Michael solrco...@gmail.com wrote: I'm using a Solr 1.4 nightly from around July. Is that recent enough to have the improved reader implementation? I'm not sure whether you'd call my operations IO heavy -- each query has so many terms (~50) that even against a 45K document index a query takes 130ms, but the entire index is in a ramfs. This could well be IO bound - lots of seeks and reads. Perhaps try on a normal filesystem and see if you still see the serialization - someone recently saw some funny results with tmpfs and lucene, so it would be good to rule that out. If you want to try and rule out tomcat, throw the webapp in jetty. -Yonik http://www.lucidimagination.com
Re: Parallel requests to Tomcat
Hi Fuad, On Wed, Sep 23, 2009 at 11:37 AM, Fuad Efendi f...@efendi.ca wrote: 8 queries against 1 Tomcat average 600ms per query, while 8 queries against 8 Tomcats average 190ms per query (on a dedicated 8 CPU server w 32G RAM). I don't see how to interpret these numbers except that Tomcat is not multithreading as well as it should :) Hi Michael, I think it is very natural; 8 single processes not sharing anything are faster than 8 threads sharing something. 8 threads sharing something may have *some* overhead versus 8 processes, but as you say, 410ms overhead points to a different problem. However, 600ms is too high. My index is on a ramfs which I've shown makes the QR and doc caches unnecessary; However, SOLR is faster than pure Lucene, try SOLR caches! I have. In a separate test, I verified that the caches that save disk I/O (QR and doc) make no difference to query time, because my index is on a ramfs. The caches that save CPU cycles (filter and fieldvalue, because I'm doing heavy faceting) DO help and I do have them turned on. Michael
Re: Parallel requests to Tomcat
Hi Yonik, On Wed, Sep 23, 2009 at 11:42 AM, Yonik Seeley yo...@lucidimagination.comwrote: This could well be IO bound - lots of seeks and reads. If this were IO bound, wouldn't I see the same results when sending my 8 requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm querying 8 processes or 8 threads in 1 process, right? Michael
Re: Parallel requests to Tomcat
This sure seems like a good time to try LucidGaze for Solr. That would give some Solr-specific profiling data. http://www.lucidimagination.com/Downloads/LucidGaze-for-Solr wunder On Sep 23, 2009, at 8:47 AM, Michael wrote: Hi Yonik, On Wed, Sep 23, 2009 at 11:42 AM, Yonik Seeley yo...@lucidimagination.comwrote: This could well be IO bound - lots of seeks and reads. If this were IO bound, wouldn't I see the same results when sending my 8 requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm querying 8 processes or 8 threads in 1 process, right? Michael
Re: Parallel requests to Tomcat
Thanks for the suggestion, Walter! I've been using Gaze 1.0 for a while now, but when I moved to a multicore approach (which was the impetus behind all of this testing) Gaze failed to start and I had to comment it out of solrconfig.xml to get Solr to start. Are you aware whether Gaze is able to work in a multicore environment? Michael On Wed, Sep 23, 2009 at 11:55 AM, Walter Underwood wun...@wunderwood.orgwrote: This sure seems like a good time to try LucidGaze for Solr. That would give some Solr-specific profiling data. http://www.lucidimagination.com/Downloads/LucidGaze-for-Solr wunder On Sep 23, 2009, at 8:47 AM, Michael wrote: Hi Yonik, On Wed, Sep 23, 2009 at 11:42 AM, Yonik Seeley yo...@lucidimagination.comwrote: This could well be IO bound - lots of seeks and reads. If this were IO bound, wouldn't I see the same results when sending my 8 requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm querying 8 processes or 8 threads in 1 process, right? Michael
Re: Parallel requests to Tomcat
On Wed, Sep 23, 2009 at 11:47 AM, Michael solrco...@gmail.com wrote: Hi Yonik, On Wed, Sep 23, 2009 at 11:42 AM, Yonik Seeley yo...@lucidimagination.com wrote: This could well be IO bound - lots of seeks and reads. If this were IO bound, wouldn't I see the same results when sending my 8 requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm querying 8 processes or 8 threads in 1 process, right? Right - I was thinking IO bound at the Lucene Directory level - which synchronized in the past and led to poor concurrency. Buy your Solr version is recent enough to use the newer unsynchronized method by default (on non-windows) -Yonik http://www.lucidimagination.com
Re: Parallel requests to Tomcat
On Wed, Sep 23, 2009 at 12:05 PM, Yonik Seeley yo...@lucidimagination.comwrote: On Wed, Sep 23, 2009 at 11:47 AM, Michael solrco...@gmail.com wrote: If this were IO bound, wouldn't I see the same results when sending my 8 requests to 8 Tomcats? There's only one disk (well, RAM) whether I'm querying 8 processes or 8 threads in 1 process, right? Right - I was thinking IO bound at the Lucene Directory level - which synchronized in the past and led to poor concurrency. Buy your Solr version is recent enough to use the newer unsynchronized method by default (on non-windows) Ah, OK. So it looks like comparing to Jetty is my only next step. Although I'm not sure what I'm going to do based on the result of that test -- if Jetty behaves differently, then I still don't know why the heck Tomcat is behaving badly! :) Michael
RE: Parallel requests to Tomcat
8 threads sharing something may have *some* overhead versus 8 processes, but as you say, 410ms overhead points to a different problem. - You have baseline (single-threaded load-stress script sending requests to SOLR) (1-request-in-parallel, 8 requests to 8 Tomcats); 200ms looks extremely high... only if you are not GETting more than top-1 docs in a single query (instead of default top-10).
RE: Parallel requests to Tomcat
I'm not sure whether you'd call my operations IO heavy -- each query has so many terms (~50) that even against a 45K document index a query takes 130ms, but the entire index is in a ramfs. The more terms, the more it takes to find docset intersections (belonging to each term); something in SOLR/Lucene is still synchronized... Try to compare with smaller 1-term queries, different terms for parallel requests...
Parallel requests to Tomcat
Hi, I have a Solr+Tomcat installation on an 8 CPU Linux box, and I just tried sending parallel requests to it and measuring response time. I would expect that it could handle up to 8 parallel requests without significant slowdown of any individual request. Instead, I found that Tomcat is serializing the requests. For example, the response time for each of 2 parallel requests is nearly 2 times that for a single request, and the time for each of 8 parallel requests is about 4 times that of a single request. I am pretty sure this is a Tomcat issue, for when I started 8 identical instances of Solr+Tomcat on the machine (on 8 different ports), I could send one request to each in parallel with only a 20% slowdown (compared to 300% in a single Tomcat.) I'm using the stock Tomcat download with minimal configuration changes, except that I disabled all logging (in case the logger was blocking for each request, serializing them.) I'm giving 2G RAM to each JVM. Does anyone more familiar with Tomcat know what's wrong? I can't imagine that Tomcat really can't handle parallel requests.
Re: Parallel requests to Tomcat
What version of Solr are you using? Solr1.3 and Lucene 2.4 defaulted to an index reader implementation that had to synchronize, so search operations that are IO heavy can't proceed in parallel. You shouldn't see this with 1.4 -Yonik http://www.lucidimagination.com On Tue, Sep 22, 2009 at 4:03 PM, Michael solrco...@gmail.com wrote: Hi, I have a Solr+Tomcat installation on an 8 CPU Linux box, and I just tried sending parallel requests to it and measuring response time. I would expect that it could handle up to 8 parallel requests without significant slowdown of any individual request. Instead, I found that Tomcat is serializing the requests. For example, the response time for each of 2 parallel requests is nearly 2 times that for a single request, and the time for each of 8 parallel requests is about 4 times that of a single request. I am pretty sure this is a Tomcat issue, for when I started 8 identical instances of Solr+Tomcat on the machine (on 8 different ports), I could send one request to each in parallel with only a 20% slowdown (compared to 300% in a single Tomcat.) I'm using the stock Tomcat download with minimal configuration changes, except that I disabled all logging (in case the logger was blocking for each request, serializing them.) I'm giving 2G RAM to each JVM. Does anyone more familiar with Tomcat know what's wrong? I can't imagine that Tomcat really can't handle parallel requests.
