I was wondering if the warmup stuff was one of the culprits (we dont have warmup's at all - the configs are pretty stock). As for the system, it seems capable of quite a bit more: memory usage is ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and load below 1.00.
The seed data and queries were put together by one of our developers. I've put all the solrmeter files here: https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure which file does which specifically. Does the system's reaction to a 'fast load' without a warmup sound normal? I would have expected the first couple hundred queries to be very slow (>500ms) and then the system catch up after a while. But it just dies very quickly and never recovers. I'll check out your SPM - I've seen it mentioned before. Thanks! -- Nate Fox Sr Systems Engineer o: 310.658.5775 m: 714.248.5350 Follow us @NEOGOV <http://twitter.com/NEOGOV> and on Facebook<http://www.facebook.com/neogov> NEOGOV <http://www.neogov.com/> is among the top fastest growing software companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and the LA Business Journal. We are hiring!<http://www.neogov.com/#/company/careers> On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > In short, certain data structures need to load from index in the > beginning, (for sorting and faceting) caches need to warm up, JVM > needs to warm up, etc., so going slowly in the beginning makes sense. > Why things die after that is a different Q. Maybe it OOMs? Maybe > queries are very complex? What do your queries look like? I see > newrelic.jar in the command-line. May want to try SPM for Solr, it > has better Solr metrics. > > Otis > -- > Solr & ElasticSearch Support > http://sematext.com/ > > > > > > On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <n...@neogov.com> wrote: > > I'm new to solr and I'm load testing our setup to see what we can handle. > > I'm using solrmeter and my problem is a bit odd: > > * When I set solrmeter to run 4000 queries/min, it will handle a few > > hundred queries and then tomcat will stop responding completely to > requests > > (even though according to lsof -i it is still listening and the java > > process is still running). > > * When I set solrmeter to run 1000 queries/min it runs fine. I can stop > > solrmeter after a couple of minutes at that pace and then run at > 4000/min > > without issue. > > > > It's as if it needs a ramp up time? Also, I noticed (regardless of ramp > up) > > that my setup cannot handle 8000/min. The reaction at 8k/min is the same > as > > if I were to run 4k/min without the ramp up. Of note, only the shard that > > solrmeter is pointed to stops responding. The other shard hums along > > without incident. > > > > Setup (everything in AWS): > > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0 > > (open-jdk-7-headless) : Ubuntu 12.04 > > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04 > > I have ~30k documents in each node (~300Mb on each node) > > > > The vast majority of my solr/tomcat7 config is default from ubuntu's > > packages/solr's example dir. Here's the configs and the end of the > > catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0 > > > > My main question is two fold: > > 1. Is this normal behavior for tomcat (to just stop responding > completely) > > when it gets overwhelmed? And the only option is to restart it? I guess I > > dont know what it looks like when tomcat/solr cant keep up. > > 2. Why does it handle better when I give it a lower number of queries and > > then ramp it up? It concerns me that if I have to restart a server in the > > cluster and it gets thrown into the pool of machines that things will > blow > > up. > > > > As an aside, does this seem like a normal amount of queries (~4k/min) > that > > this kind of environment should be able to handle? >
