Hi Nate; This may be out of topic however could you explain that why you want to use Tomcat instead of Jetty or Embedded Jetty?
2013/3/27 Michael Della Bitta <michael.della.bi...@appinions.com> > You're using the blocking IO connector, which isn't so great for heavy > loads. > > Give this a shot... You'll end up with 8192 max connections by > default, although this is tunable too: > > Run: > apt-get install libapr1 libtcnative-1 > > Add this to the list of Listeners at the top of server.xml: > > <Listener className="org.apache.catalina.core.AprLifecycleListener" > SSLEngine="off" /> > > These instructions assume you're running Tomcat 6 or 7. > > Here's some documentation: > http://tomcat.apache.org/tomcat-7.0-doc/apr.html > http://tomcat.apache.org/tomcat-7.0-doc/config/http.html > > > Michael Della Bitta > > ------------------------------------------------ > Appinions > 18 East 41st Street, 2nd Floor > New York, NY 10017-6271 > > www.appinions.com > > Where Influence Isn’t a Game > > > On Tue, Mar 26, 2013 at 5:31 PM, Nate Fox <n...@neogov.com> wrote: > > We're not using ELB and I have no idea which connector I'm using - I'm > > guessing whatever is default (I'm a total noob). This is from my > server.xml: > > <Connector port="8080" protocol="HTTP/1.1" > connectionTimeout="60000" > > URIEncoding="UTF-8" redirectPort="8443" /> > > > > > > > > -- > > Nate Fox > > Sr Systems Engineer > > > > o: 310.658.5775 > > m: 714.248.5350 > > > > Follow us @NEOGOV <http://twitter.com/NEOGOV> and on > > Facebook<http://www.facebook.com/neogov> > > > > NEOGOV <http://www.neogov.com/> is among the top fastest growing > software > > companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and > > the LA Business Journal. We are hiring!< > http://www.neogov.com/#/company/careers> > > > > > > > > On Tue, Mar 26, 2013 at 1:02 PM, Michael Della Bitta < > > michael.della.bi...@appinions.com> wrote: > > > >> Nate, > >> > >> We just cleared up a problem similar to this by ditching Elastic Load > >> Balancer and switching over to the APR connector in Tomcat. Are you > >> using either of those? > >> > >> Michael Della Bitta > >> > >> ------------------------------------------------ > >> Appinions > >> 18 East 41st Street, 2nd Floor > >> New York, NY 10017-6271 > >> > >> www.appinions.com > >> > >> Where Influence Isn’t a Game > >> > >> > >> On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic > >> <otis.gospodne...@gmail.com> wrote: > >> > Hi Nate, > >> > > >> > Try adding some warmup queries and making sure the setting for using > >> > the cold searcher in solrconfig.xml is set to false. Your warmup > >> > queries should use facets and sorting if your normal queries use them. > >> > In SPM you'll actually see how much time warming up takes, so you'll > >> > get a better idea of the "cost" of that (when you don't do it). > >> > > >> > Otis > >> > -- > >> > Solr & ElasticSearch Support > >> > http://sematext.com/ > >> > > >> > > >> > > >> > > >> > > >> > On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox <n...@neogov.com> wrote: > >> >> I was wondering if the warmup stuff was one of the culprits (we dont > >> have > >> >> warmup's at all - the configs are pretty stock). > >> >> As for the system, it seems capable of quite a bit more: memory > usage is > >> >> ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) > >> and > >> >> load below 1.00. > >> >> > >> >> The seed data and queries were put together by one of our developers. > >> I've > >> >> put all the solrmeter files here: > >> >> https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce > >> >> Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely > >> sure > >> >> which file does which specifically. > >> >> > >> >> Does the system's reaction to a 'fast load' without a warmup sound > >> normal? > >> >> I would have expected the first couple hundred queries to be very > slow > >> >> (>500ms) and then the system catch up after a while. But it just dies > >> very > >> >> quickly and never recovers. > >> >> > >> >> I'll check out your SPM - I've seen it mentioned before. Thanks! > >> >> > >> >> > >> >> > >> >> -- > >> >> Nate Fox > >> >> Sr Systems Engineer > >> >> > >> >> o: 310.658.5775 > >> >> m: 714.248.5350 > >> >> > >> >> Follow us @NEOGOV <http://twitter.com/NEOGOV> and on > >> >> Facebook<http://www.facebook.com/neogov> > >> >> > >> >> NEOGOV <http://www.neogov.com/> is among the top fastest growing > >> software > >> >> companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, > and > >> >> the LA Business Journal. We are hiring!< > >> http://www.neogov.com/#/company/careers> > >> >> > >> >> > >> >> > >> >> On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic < > >> >> otis.gospodne...@gmail.com> wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> In short, certain data structures need to load from index in the > >> >>> beginning, (for sorting and faceting) caches need to warm up, JVM > >> >>> needs to warm up, etc., so going slowly in the beginning makes > sense. > >> >>> Why things die after that is a different Q. Maybe it OOMs? Maybe > >> >>> queries are very complex? What do your queries look like? I see > >> >>> newrelic.jar in the command-line. May want to try SPM for Solr, it > >> >>> has better Solr metrics. > >> >>> > >> >>> Otis > >> >>> -- > >> >>> Solr & ElasticSearch Support > >> >>> http://sematext.com/ > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <n...@neogov.com> wrote: > >> >>> > I'm new to solr and I'm load testing our setup to see what we can > >> handle. > >> >>> > I'm using solrmeter and my problem is a bit odd: > >> >>> > * When I set solrmeter to run 4000 queries/min, it will handle a > few > >> >>> > hundred queries and then tomcat will stop responding completely to > >> >>> requests > >> >>> > (even though according to lsof -i it is still listening and the > java > >> >>> > process is still running). > >> >>> > * When I set solrmeter to run 1000 queries/min it runs fine. I can > >> stop > >> >>> > solrmeter after a couple of minutes at that pace and then run at > >> >>> 4000/min > >> >>> > without issue. > >> >>> > > >> >>> > It's as if it needs a ramp up time? Also, I noticed (regardless of > >> ramp > >> >>> up) > >> >>> > that my setup cannot handle 8000/min. The reaction at 8k/min is > the > >> same > >> >>> as > >> >>> > if I were to run 4k/min without the ramp up. Of note, only the > shard > >> that > >> >>> > solrmeter is pointed to stops responding. The other shard hums > along > >> >>> > without incident. > >> >>> > > >> >>> > Setup (everything in AWS): > >> >>> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0 > >> >>> > (open-jdk-7-headless) : Ubuntu 12.04 > >> >>> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04 > >> >>> > I have ~30k documents in each node (~300Mb on each node) > >> >>> > > >> >>> > The vast majority of my solr/tomcat7 config is default from > ubuntu's > >> >>> > packages/solr's example dir. Here's the configs and the end of the > >> >>> > catalina.out file: > >> https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0 > >> >>> > > >> >>> > My main question is two fold: > >> >>> > 1. Is this normal behavior for tomcat (to just stop responding > >> >>> completely) > >> >>> > when it gets overwhelmed? And the only option is to restart it? I > >> guess I > >> >>> > dont know what it looks like when tomcat/solr cant keep up. > >> >>> > 2. Why does it handle better when I give it a lower number of > >> queries and > >> >>> > then ramp it up? It concerns me that if I have to restart a server > >> in the > >> >>> > cluster and it gets thrown into the pool of machines that things > will > >> >>> blow > >> >>> > up. > >> >>> > > >> >>> > As an aside, does this seem like a normal amount of queries > (~4k/min) > >> >>> that > >> >>> > this kind of environment should be able to handle? > >> >>> > >> >
