5 min is ridiculously long for a query that used to take 65ms. That ought to be a great clue. The only two things I've seen that could cause that are thrashing, or GC. Hard to see how it could be thrashing, given your hardware, so I'd initially suspect GC.
Aim VisualVM at the JVM. It shows how much CPU goes to GC over time, in a nice blue line. And if it's not GC, try out its Sampler tab, and see where the CPU is spending its time. FWIW, when asked at what point one would want to split JVMs and shard, on the same machine, Grant Ingersoll mentioned 16GB, and precisely for GC cost reasons. You're way above that. Maybe multiple JVMs and sharding, even on the same machine, would serve you better than a monster 70GB JVM. -- Bryan > -----Original Message----- > From: Mou [mailto:mouna...@gmail.com] > Sent: Monday, July 16, 2012 7:43 PM > To: solr-user@lucene.apache.org > Subject: Using Solr 3.4 running on tomcat7 - very slow search > > Hi, > > Our index is divided into two shards and each of them has 120M docs , > total > size 75G in each core. > The server is a pretty good one , jvm is given memory of 70G and about > same > is left for OS (SLES 11) . > > We use all dynamic fields except th eunique id and are using long queries > but almost all of them are filter queires, Each query may have 10 -30 fq > parameters. > > When I tested the index ( same size) but with max heap size 40 G, queries > were blazing fast. I used solrmeter to load test and it was happily > serving > 12000 queries or more per min with avg 65 ms qtime.We had an excellent > filtercache hit ratio. > > This index is only used for searching and being replicated every 7 sec > from > the master. > > But now in production server it is horribly slow and taking 5 mins(qtime) > to > return a query ( same query). > What could go wrong? > > Really appreciate your suggestions on debugging this thing.. > > > > -- > View this message in context: http://lucene.472066.n3.nabble.com/Using- > Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436.html > Sent from the Solr - User mailing list archive at Nabble.com.
