Hi Nate, Try adding some warmup queries and making sure the setting for using the cold searcher in solrconfig.xml is set to false. Your warmup queries should use facets and sorting if your normal queries use them. In SPM you'll actually see how much time warming up takes, so you'll get a better idea of the "cost" of that (when you don't do it).
Otis -- Solr & ElasticSearch Support http://sematext.com/ On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox <n...@neogov.com> wrote: > I was wondering if the warmup stuff was one of the culprits (we dont have > warmup's at all - the configs are pretty stock). > As for the system, it seems capable of quite a bit more: memory usage is > ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and > load below 1.00. > > The seed data and queries were put together by one of our developers. I've > put all the solrmeter files here: > https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce > Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure > which file does which specifically. > > Does the system's reaction to a 'fast load' without a warmup sound normal? > I would have expected the first couple hundred queries to be very slow > (>500ms) and then the system catch up after a while. But it just dies very > quickly and never recovers. > > I'll check out your SPM - I've seen it mentioned before. Thanks! > > > > -- > Nate Fox > Sr Systems Engineer > > o: 310.658.5775 > m: 714.248.5350 > > Follow us @NEOGOV <http://twitter.com/NEOGOV> and on > Facebook<http://www.facebook.com/neogov> > > NEOGOV <http://www.neogov.com/> is among the top fastest growing software > companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and > the LA Business Journal. We are > hiring!<http://www.neogov.com/#/company/careers> > > > > On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > >> Hi, >> >> In short, certain data structures need to load from index in the >> beginning, (for sorting and faceting) caches need to warm up, JVM >> needs to warm up, etc., so going slowly in the beginning makes sense. >> Why things die after that is a different Q. Maybe it OOMs? Maybe >> queries are very complex? What do your queries look like? I see >> newrelic.jar in the command-line. May want to try SPM for Solr, it >> has better Solr metrics. >> >> Otis >> -- >> Solr & ElasticSearch Support >> http://sematext.com/ >> >> >> >> >> >> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <n...@neogov.com> wrote: >> > I'm new to solr and I'm load testing our setup to see what we can handle. >> > I'm using solrmeter and my problem is a bit odd: >> > * When I set solrmeter to run 4000 queries/min, it will handle a few >> > hundred queries and then tomcat will stop responding completely to >> requests >> > (even though according to lsof -i it is still listening and the java >> > process is still running). >> > * When I set solrmeter to run 1000 queries/min it runs fine. I can stop >> > solrmeter after a couple of minutes at that pace and then run at >> 4000/min >> > without issue. >> > >> > It's as if it needs a ramp up time? Also, I noticed (regardless of ramp >> up) >> > that my setup cannot handle 8000/min. The reaction at 8k/min is the same >> as >> > if I were to run 4k/min without the ramp up. Of note, only the shard that >> > solrmeter is pointed to stops responding. The other shard hums along >> > without incident. >> > >> > Setup (everything in AWS): >> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0 >> > (open-jdk-7-headless) : Ubuntu 12.04 >> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04 >> > I have ~30k documents in each node (~300Mb on each node) >> > >> > The vast majority of my solr/tomcat7 config is default from ubuntu's >> > packages/solr's example dir. Here's the configs and the end of the >> > catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0 >> > >> > My main question is two fold: >> > 1. Is this normal behavior for tomcat (to just stop responding >> completely) >> > when it gets overwhelmed? And the only option is to restart it? I guess I >> > dont know what it looks like when tomcat/solr cant keep up. >> > 2. Why does it handle better when I give it a lower number of queries and >> > then ramp it up? It concerns me that if I have to restart a server in the >> > cluster and it gets thrown into the pool of machines that things will >> blow >> > up. >> > >> > As an aside, does this seem like a normal amount of queries (~4k/min) >> that >> > this kind of environment should be able to handle? >>
