Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Hi Nate; This may be out of topic however could you explain that why you want to use Tomcat instead of Jetty or Embedded Jetty? 2013/3/27 Michael Della Bitta michael.della.bi...@appinions.com You're using the blocking IO connector, which isn't so great for heavy loads. Give this a shot... You'll end up with 8192 max connections by default, although this is tunable too: Run: apt-get install libapr1 libtcnative-1 Add this to the list of Listeners at the top of server.xml: Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=off / These instructions assume you're running Tomcat 6 or 7. Here's some documentation: http://tomcat.apache.org/tomcat-7.0-doc/apr.html http://tomcat.apache.org/tomcat-7.0-doc/config/http.html Michael Della Bitta Appinions 18 East 41st Street, 2nd Floor New York, NY 10017-6271 www.appinions.com Where Influence Isn’t a Game On Tue, Mar 26, 2013 at 5:31 PM, Nate Fox n...@neogov.com wrote: We're not using ELB and I have no idea which connector I'm using - I'm guessing whatever is default (I'm a total noob). This is from my server.xml: Connector port=8080 protocol=HTTP/1.1 connectionTimeout=6 URIEncoding=UTF-8 redirectPort=8443 / -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring! http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 1:02 PM, Michael Della Bitta michael.della.bi...@appinions.com wrote: Nate, We just cleared up a problem similar to this by ditching Elastic Load Balancer and switching over to the APR connector in Tomcat. Are you using either of those? Michael Della Bitta Appinions 18 East 41st Street, 2nd Floor New York, NY 10017-6271 www.appinions.com Where Influence Isn’t a Game On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi Nate, Try adding some warmup queries and making sure the setting for using the cold searcher in solrconfig.xml is set to false. Your warmup queries should use facets and sorting if your normal queries use them. In SPM you'll actually see how much time warming up takes, so you'll get a better idea of the cost of that (when you don't do it). Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox n...@neogov.com wrote: I was wondering if the warmup stuff was one of the culprits (we dont have warmup's at all - the configs are pretty stock). As for the system, it seems capable of quite a bit more: memory usage is ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and load below 1.00. The seed data and queries were put together by one of our developers. I've put all the solrmeter files here: https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure which file does which specifically. Does the system's reaction to a 'fast load' without a warmup sound normal? I would have expected the first couple hundred queries to be very slow (500ms) and then the system catch up after a while. But it just dies very quickly and never recovers. I'll check out your SPM - I've seen it mentioned before. Thanks! -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring! http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, In short, certain data structures need to load from index in the beginning, (for sorting and faceting) caches need to warm up, JVM needs to warm up, etc., so going slowly in the beginning makes sense. Why things die after that is a different Q. Maybe it OOMs? Maybe queries are very complex? What do your queries look like? I see newrelic.jar in the command-line. May want to try SPM for Solr, it has better Solr metrics. Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox n...@neogov.com wrote: I'm new to solr and I'm load testing our setup to see what we can handle.
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Update: issue resolved! Cranking up the maxThreads did the trick. Default is 200. I went with 2500 for grins and giggles and things work great. Now, even if I overwhelm the box with too many requests, when the requests back off the box continues to respond. And when I slam the server after it's been restarted (without having warmup queries), it acts as I wanted: queries are slow to respond (upwards of 30s) for the first couple minutes then they start to all be under 25ms and normalize at a very fast pace (obviously as the cache is warmed). Christopher, I could have sworn I tried upping acceptCount, maxConnections and maxThreads in my testing, but with your prodding I tried it again - and that was the solution. I have a couple quick followup questions: - What is the downside of having a maxThreads, acceptCount and maxConnections really high? Obviously defaults are there for a reason - I'd like to know what the reasoning is. - Any reason I shouldnt use Tomcat? I just went with it because I figured it was extremely mature and was easy to use with apt-get :) I'll probably toy with the APR as suggested by Michael, as I like the idea of a non-blocking connector. -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring!http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 5:56 PM, Chris Hostetter hossman_luc...@fucit.orgwrote: : * When I set solrmeter to run 4000 queries/min, it will handle a few : hundred queries and then tomcat will stop responding completely to requests : (even though according to lsof -i it is still listening and the java : process is still running). have you tried tacking using jstack to generate a thread dump of the server to see what it's doing? : * When I set solrmeter to run 1000 queries/min it runs fine. I can stop : solrmeter after a couple of minutes at that pace and then run at 4000/min : without issue. : : It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up) : that my setup cannot handle 8000/min. The reaction at 8k/min is the same as : if I were to run 4k/min without the ramp up. Of note, only the shard that : solrmeter is pointed to stops responding. The other shard hums along : without incident. Just to clarify: you're running a 2 node SolrCloud cluster, where each node contains a unique shard, and pointing solrmeter at a single node for the queries -- correct? Here's my hunch: you are probably hitting the limit of the number of concurrent connections tomcat will allow (whatever it may be confiurged ot in your setup). In the 8000/min case, you are probably maxing out that limit with direct connections you issue from solrmeter to that single node. In the 4000/min case, each request you issue causes that single node to fire off multiple requests to each shard, and since each shard exists on only one node, you are garunteeing thta you double the number of concurrent requests hitting that first node. in the case where you start w/ 1000/min, and then later ramp up to 4000/min, you are probably causing enough of the queries to be warmed up that they are in the caches on both nodes, so they can be served really fast and return their results before you reach that max number of concurrent connections after you ramp up. I'm no tomcat expert, but skimming hte docs, you may want to look at settings like acceptCount, maxConnections, maxThreads, etc... -Hoss
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
On 3/27/2013 1:16 PM, Nate Fox wrote: I have a couple quick followup questions: - What is the downside of having a maxThreads, acceptCount and maxConnections really high? Obviously defaults are there for a reason - I'd like to know what the reasoning is. - Any reason I shouldnt use Tomcat? I just went with it because I figured it was extremely mature and was easy to use with apt-get :) The maxThreads parameter in the jetty config that's included with Solr is set to 1 - this is the value chosen by Solr's development team. Your setting of 2500 should be perfectly fine, and it is definitely not really high. The default of 200 in your distribution is very low. Tomcat is certainly a viable solution, one used by many. It is very mature and has proven itself. The really nice thing with using an OS-packaged version is that you don't have to write or change the init script. I use the jetty that was included with Solr, and had to write my own init script. Jetty, especially the stripped-down version included with Solr, has a smaller footprint than tomcat. The bells and whistles are not required. It is not better or worse than tomcat, just another choice. Thanks, Shawn
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
On Mar 27, 2013, at 3:29 PM, Shawn Heisey s...@elyograg.org wrote: The maxThreads parameter in the jetty config that's included with Solr is set to 1 Yonik raised this at some point if I remember right - it helps avoid some distrib deadlock issue. - Mark
Loadtesting solr/tomcat7 and tomcat stops responding entirely
I'm new to solr and I'm load testing our setup to see what we can handle. I'm using solrmeter and my problem is a bit odd: * When I set solrmeter to run 4000 queries/min, it will handle a few hundred queries and then tomcat will stop responding completely to requests (even though according to lsof -i it is still listening and the java process is still running). * When I set solrmeter to run 1000 queries/min it runs fine. I can stop solrmeter after a couple of minutes at that pace and then run at 4000/min without issue. It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up) that my setup cannot handle 8000/min. The reaction at 8k/min is the same as if I were to run 4k/min without the ramp up. Of note, only the shard that solrmeter is pointed to stops responding. The other shard hums along without incident. Setup (everything in AWS): - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0 (open-jdk-7-headless) : Ubuntu 12.04 - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04 I have ~30k documents in each node (~300Mb on each node) The vast majority of my solr/tomcat7 config is default from ubuntu's packages/solr's example dir. Here's the configs and the end of the catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0 My main question is two fold: 1. Is this normal behavior for tomcat (to just stop responding completely) when it gets overwhelmed? And the only option is to restart it? I guess I dont know what it looks like when tomcat/solr cant keep up. 2. Why does it handle better when I give it a lower number of queries and then ramp it up? It concerns me that if I have to restart a server in the cluster and it gets thrown into the pool of machines that things will blow up. As an aside, does this seem like a normal amount of queries (~4k/min) that this kind of environment should be able to handle?
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Hi, In short, certain data structures need to load from index in the beginning, (for sorting and faceting) caches need to warm up, JVM needs to warm up, etc., so going slowly in the beginning makes sense. Why things die after that is a different Q. Maybe it OOMs? Maybe queries are very complex? What do your queries look like? I see newrelic.jar in the command-line. May want to try SPM for Solr, it has better Solr metrics. Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox n...@neogov.com wrote: I'm new to solr and I'm load testing our setup to see what we can handle. I'm using solrmeter and my problem is a bit odd: * When I set solrmeter to run 4000 queries/min, it will handle a few hundred queries and then tomcat will stop responding completely to requests (even though according to lsof -i it is still listening and the java process is still running). * When I set solrmeter to run 1000 queries/min it runs fine. I can stop solrmeter after a couple of minutes at that pace and then run at 4000/min without issue. It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up) that my setup cannot handle 8000/min. The reaction at 8k/min is the same as if I were to run 4k/min without the ramp up. Of note, only the shard that solrmeter is pointed to stops responding. The other shard hums along without incident. Setup (everything in AWS): - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0 (open-jdk-7-headless) : Ubuntu 12.04 - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04 I have ~30k documents in each node (~300Mb on each node) The vast majority of my solr/tomcat7 config is default from ubuntu's packages/solr's example dir. Here's the configs and the end of the catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0 My main question is two fold: 1. Is this normal behavior for tomcat (to just stop responding completely) when it gets overwhelmed? And the only option is to restart it? I guess I dont know what it looks like when tomcat/solr cant keep up. 2. Why does it handle better when I give it a lower number of queries and then ramp it up? It concerns me that if I have to restart a server in the cluster and it gets thrown into the pool of machines that things will blow up. As an aside, does this seem like a normal amount of queries (~4k/min) that this kind of environment should be able to handle?
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
I was wondering if the warmup stuff was one of the culprits (we dont have warmup's at all - the configs are pretty stock). As for the system, it seems capable of quite a bit more: memory usage is ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and load below 1.00. The seed data and queries were put together by one of our developers. I've put all the solrmeter files here: https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure which file does which specifically. Does the system's reaction to a 'fast load' without a warmup sound normal? I would have expected the first couple hundred queries to be very slow (500ms) and then the system catch up after a while. But it just dies very quickly and never recovers. I'll check out your SPM - I've seen it mentioned before. Thanks! -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring!http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, In short, certain data structures need to load from index in the beginning, (for sorting and faceting) caches need to warm up, JVM needs to warm up, etc., so going slowly in the beginning makes sense. Why things die after that is a different Q. Maybe it OOMs? Maybe queries are very complex? What do your queries look like? I see newrelic.jar in the command-line. May want to try SPM for Solr, it has better Solr metrics. Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox n...@neogov.com wrote: I'm new to solr and I'm load testing our setup to see what we can handle. I'm using solrmeter and my problem is a bit odd: * When I set solrmeter to run 4000 queries/min, it will handle a few hundred queries and then tomcat will stop responding completely to requests (even though according to lsof -i it is still listening and the java process is still running). * When I set solrmeter to run 1000 queries/min it runs fine. I can stop solrmeter after a couple of minutes at that pace and then run at 4000/min without issue. It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up) that my setup cannot handle 8000/min. The reaction at 8k/min is the same as if I were to run 4k/min without the ramp up. Of note, only the shard that solrmeter is pointed to stops responding. The other shard hums along without incident. Setup (everything in AWS): - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0 (open-jdk-7-headless) : Ubuntu 12.04 - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04 I have ~30k documents in each node (~300Mb on each node) The vast majority of my solr/tomcat7 config is default from ubuntu's packages/solr's example dir. Here's the configs and the end of the catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0 My main question is two fold: 1. Is this normal behavior for tomcat (to just stop responding completely) when it gets overwhelmed? And the only option is to restart it? I guess I dont know what it looks like when tomcat/solr cant keep up. 2. Why does it handle better when I give it a lower number of queries and then ramp it up? It concerns me that if I have to restart a server in the cluster and it gets thrown into the pool of machines that things will blow up. As an aside, does this seem like a normal amount of queries (~4k/min) that this kind of environment should be able to handle?
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Hi Nate, Try adding some warmup queries and making sure the setting for using the cold searcher in solrconfig.xml is set to false. Your warmup queries should use facets and sorting if your normal queries use them. In SPM you'll actually see how much time warming up takes, so you'll get a better idea of the cost of that (when you don't do it). Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox n...@neogov.com wrote: I was wondering if the warmup stuff was one of the culprits (we dont have warmup's at all - the configs are pretty stock). As for the system, it seems capable of quite a bit more: memory usage is ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and load below 1.00. The seed data and queries were put together by one of our developers. I've put all the solrmeter files here: https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure which file does which specifically. Does the system's reaction to a 'fast load' without a warmup sound normal? I would have expected the first couple hundred queries to be very slow (500ms) and then the system catch up after a while. But it just dies very quickly and never recovers. I'll check out your SPM - I've seen it mentioned before. Thanks! -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring!http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, In short, certain data structures need to load from index in the beginning, (for sorting and faceting) caches need to warm up, JVM needs to warm up, etc., so going slowly in the beginning makes sense. Why things die after that is a different Q. Maybe it OOMs? Maybe queries are very complex? What do your queries look like? I see newrelic.jar in the command-line. May want to try SPM for Solr, it has better Solr metrics. Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox n...@neogov.com wrote: I'm new to solr and I'm load testing our setup to see what we can handle. I'm using solrmeter and my problem is a bit odd: * When I set solrmeter to run 4000 queries/min, it will handle a few hundred queries and then tomcat will stop responding completely to requests (even though according to lsof -i it is still listening and the java process is still running). * When I set solrmeter to run 1000 queries/min it runs fine. I can stop solrmeter after a couple of minutes at that pace and then run at 4000/min without issue. It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up) that my setup cannot handle 8000/min. The reaction at 8k/min is the same as if I were to run 4k/min without the ramp up. Of note, only the shard that solrmeter is pointed to stops responding. The other shard hums along without incident. Setup (everything in AWS): - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0 (open-jdk-7-headless) : Ubuntu 12.04 - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04 I have ~30k documents in each node (~300Mb on each node) The vast majority of my solr/tomcat7 config is default from ubuntu's packages/solr's example dir. Here's the configs and the end of the catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0 My main question is two fold: 1. Is this normal behavior for tomcat (to just stop responding completely) when it gets overwhelmed? And the only option is to restart it? I guess I dont know what it looks like when tomcat/solr cant keep up. 2. Why does it handle better when I give it a lower number of queries and then ramp it up? It concerns me that if I have to restart a server in the cluster and it gets thrown into the pool of machines that things will blow up. As an aside, does this seem like a normal amount of queries (~4k/min) that this kind of environment should be able to handle?
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Nate, We just cleared up a problem similar to this by ditching Elastic Load Balancer and switching over to the APR connector in Tomcat. Are you using either of those? Michael Della Bitta Appinions 18 East 41st Street, 2nd Floor New York, NY 10017-6271 www.appinions.com Where Influence Isn’t a Game On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi Nate, Try adding some warmup queries and making sure the setting for using the cold searcher in solrconfig.xml is set to false. Your warmup queries should use facets and sorting if your normal queries use them. In SPM you'll actually see how much time warming up takes, so you'll get a better idea of the cost of that (when you don't do it). Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox n...@neogov.com wrote: I was wondering if the warmup stuff was one of the culprits (we dont have warmup's at all - the configs are pretty stock). As for the system, it seems capable of quite a bit more: memory usage is ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and load below 1.00. The seed data and queries were put together by one of our developers. I've put all the solrmeter files here: https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure which file does which specifically. Does the system's reaction to a 'fast load' without a warmup sound normal? I would have expected the first couple hundred queries to be very slow (500ms) and then the system catch up after a while. But it just dies very quickly and never recovers. I'll check out your SPM - I've seen it mentioned before. Thanks! -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring!http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, In short, certain data structures need to load from index in the beginning, (for sorting and faceting) caches need to warm up, JVM needs to warm up, etc., so going slowly in the beginning makes sense. Why things die after that is a different Q. Maybe it OOMs? Maybe queries are very complex? What do your queries look like? I see newrelic.jar in the command-line. May want to try SPM for Solr, it has better Solr metrics. Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox n...@neogov.com wrote: I'm new to solr and I'm load testing our setup to see what we can handle. I'm using solrmeter and my problem is a bit odd: * When I set solrmeter to run 4000 queries/min, it will handle a few hundred queries and then tomcat will stop responding completely to requests (even though according to lsof -i it is still listening and the java process is still running). * When I set solrmeter to run 1000 queries/min it runs fine. I can stop solrmeter after a couple of minutes at that pace and then run at 4000/min without issue. It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up) that my setup cannot handle 8000/min. The reaction at 8k/min is the same as if I were to run 4k/min without the ramp up. Of note, only the shard that solrmeter is pointed to stops responding. The other shard hums along without incident. Setup (everything in AWS): - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0 (open-jdk-7-headless) : Ubuntu 12.04 - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04 I have ~30k documents in each node (~300Mb on each node) The vast majority of my solr/tomcat7 config is default from ubuntu's packages/solr's example dir. Here's the configs and the end of the catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0 My main question is two fold: 1. Is this normal behavior for tomcat (to just stop responding completely) when it gets overwhelmed? And the only option is to restart it? I guess I dont know what it looks like when tomcat/solr cant keep up. 2. Why does it handle better when I give it a lower number of queries and then ramp it up? It concerns me that if I have to restart a server in the cluster and it gets thrown into the pool of machines that things will blow up. As an aside, does this seem like a normal amount of queries (~4k/min) that this kind of environment should be able to handle?
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
We're not using ELB and I have no idea which connector I'm using - I'm guessing whatever is default (I'm a total noob). This is from my server.xml: Connector port=8080 protocol=HTTP/1.1 connectionTimeout=6 URIEncoding=UTF-8 redirectPort=8443 / -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring!http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 1:02 PM, Michael Della Bitta michael.della.bi...@appinions.com wrote: Nate, We just cleared up a problem similar to this by ditching Elastic Load Balancer and switching over to the APR connector in Tomcat. Are you using either of those? Michael Della Bitta Appinions 18 East 41st Street, 2nd Floor New York, NY 10017-6271 www.appinions.com Where Influence Isn’t a Game On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi Nate, Try adding some warmup queries and making sure the setting for using the cold searcher in solrconfig.xml is set to false. Your warmup queries should use facets and sorting if your normal queries use them. In SPM you'll actually see how much time warming up takes, so you'll get a better idea of the cost of that (when you don't do it). Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox n...@neogov.com wrote: I was wondering if the warmup stuff was one of the culprits (we dont have warmup's at all - the configs are pretty stock). As for the system, it seems capable of quite a bit more: memory usage is ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and load below 1.00. The seed data and queries were put together by one of our developers. I've put all the solrmeter files here: https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure which file does which specifically. Does the system's reaction to a 'fast load' without a warmup sound normal? I would have expected the first couple hundred queries to be very slow (500ms) and then the system catch up after a while. But it just dies very quickly and never recovers. I'll check out your SPM - I've seen it mentioned before. Thanks! -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring! http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, In short, certain data structures need to load from index in the beginning, (for sorting and faceting) caches need to warm up, JVM needs to warm up, etc., so going slowly in the beginning makes sense. Why things die after that is a different Q. Maybe it OOMs? Maybe queries are very complex? What do your queries look like? I see newrelic.jar in the command-line. May want to try SPM for Solr, it has better Solr metrics. Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox n...@neogov.com wrote: I'm new to solr and I'm load testing our setup to see what we can handle. I'm using solrmeter and my problem is a bit odd: * When I set solrmeter to run 4000 queries/min, it will handle a few hundred queries and then tomcat will stop responding completely to requests (even though according to lsof -i it is still listening and the java process is still running). * When I set solrmeter to run 1000 queries/min it runs fine. I can stop solrmeter after a couple of minutes at that pace and then run at 4000/min without issue. It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up) that my setup cannot handle 8000/min. The reaction at 8k/min is the same as if I were to run 4k/min without the ramp up. Of note, only the shard that solrmeter is pointed to stops responding. The other shard hums along without incident. Setup (everything in AWS): - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0 (open-jdk-7-headless) : Ubuntu 12.04 - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04 I have ~30k documents in each node (~300Mb on each node) The vast majority of my solr/tomcat7 config is default from ubuntu's packages/solr's example dir. Here's the configs
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
: * When I set solrmeter to run 4000 queries/min, it will handle a few : hundred queries and then tomcat will stop responding completely to requests : (even though according to lsof -i it is still listening and the java : process is still running). have you tried tacking using jstack to generate a thread dump of the server to see what it's doing? : * When I set solrmeter to run 1000 queries/min it runs fine. I can stop : solrmeter after a couple of minutes at that pace and then run at 4000/min : without issue. : : It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up) : that my setup cannot handle 8000/min. The reaction at 8k/min is the same as : if I were to run 4k/min without the ramp up. Of note, only the shard that : solrmeter is pointed to stops responding. The other shard hums along : without incident. Just to clarify: you're running a 2 node SolrCloud cluster, where each node contains a unique shard, and pointing solrmeter at a single node for the queries -- correct? Here's my hunch: you are probably hitting the limit of the number of concurrent connections tomcat will allow (whatever it may be confiurged ot in your setup). In the 8000/min case, you are probably maxing out that limit with direct connections you issue from solrmeter to that single node. In the 4000/min case, each request you issue causes that single node to fire off multiple requests to each shard, and since each shard exists on only one node, you are garunteeing thta you double the number of concurrent requests hitting that first node. in the case where you start w/ 1000/min, and then later ramp up to 4000/min, you are probably causing enough of the queries to be warmed up that they are in the caches on both nodes, so they can be served really fast and return their results before you reach that max number of concurrent connections after you ramp up. I'm no tomcat expert, but skimming hte docs, you may want to look at settings like acceptCount, maxConnections, maxThreads, etc... -Hoss
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
You're using the blocking IO connector, which isn't so great for heavy loads. Give this a shot... You'll end up with 8192 max connections by default, although this is tunable too: Run: apt-get install libapr1 libtcnative-1 Add this to the list of Listeners at the top of server.xml: Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=off / These instructions assume you're running Tomcat 6 or 7. Here's some documentation: http://tomcat.apache.org/tomcat-7.0-doc/apr.html http://tomcat.apache.org/tomcat-7.0-doc/config/http.html Michael Della Bitta Appinions 18 East 41st Street, 2nd Floor New York, NY 10017-6271 www.appinions.com Where Influence Isn’t a Game On Tue, Mar 26, 2013 at 5:31 PM, Nate Fox n...@neogov.com wrote: We're not using ELB and I have no idea which connector I'm using - I'm guessing whatever is default (I'm a total noob). This is from my server.xml: Connector port=8080 protocol=HTTP/1.1 connectionTimeout=6 URIEncoding=UTF-8 redirectPort=8443 / -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring!http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 1:02 PM, Michael Della Bitta michael.della.bi...@appinions.com wrote: Nate, We just cleared up a problem similar to this by ditching Elastic Load Balancer and switching over to the APR connector in Tomcat. Are you using either of those? Michael Della Bitta Appinions 18 East 41st Street, 2nd Floor New York, NY 10017-6271 www.appinions.com Where Influence Isn’t a Game On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi Nate, Try adding some warmup queries and making sure the setting for using the cold searcher in solrconfig.xml is set to false. Your warmup queries should use facets and sorting if your normal queries use them. In SPM you'll actually see how much time warming up takes, so you'll get a better idea of the cost of that (when you don't do it). Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox n...@neogov.com wrote: I was wondering if the warmup stuff was one of the culprits (we dont have warmup's at all - the configs are pretty stock). As for the system, it seems capable of quite a bit more: memory usage is ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and load below 1.00. The seed data and queries were put together by one of our developers. I've put all the solrmeter files here: https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure which file does which specifically. Does the system's reaction to a 'fast load' without a warmup sound normal? I would have expected the first couple hundred queries to be very slow (500ms) and then the system catch up after a while. But it just dies very quickly and never recovers. I'll check out your SPM - I've seen it mentioned before. Thanks! -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV http://twitter.com/NEOGOV and on Facebookhttp://www.facebook.com/neogov NEOGOV http://www.neogov.com/ is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring! http://www.neogov.com/#/company/careers On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, In short, certain data structures need to load from index in the beginning, (for sorting and faceting) caches need to warm up, JVM needs to warm up, etc., so going slowly in the beginning makes sense. Why things die after that is a different Q. Maybe it OOMs? Maybe queries are very complex? What do your queries look like? I see newrelic.jar in the command-line. May want to try SPM for Solr, it has better Solr metrics. Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox n...@neogov.com wrote: I'm new to solr and I'm load testing our setup to see what we can handle. I'm using solrmeter and my problem is a bit odd: * When I set solrmeter to run 4000 queries/min, it will handle a few hundred queries and then tomcat will stop responding completely to requests (even though according to lsof -i it is still listening and the java process is still running). * When I set solrmeter to run 1000 queries/min it
