Brian, Thanks again. swappiness is set to 60 and from vmstat , I can see no swapping is going on. Also I am using fusion IO SSD for storing my index.
I also used the visualVM and it shows me that it is blocked on lock=org.apache.lucene.index.SegmentCoreReaders@299172a7. Any clue? On Mon, Jul 16, 2012 at 10:38 PM, Bryan Loofbourrow [via Lucene] < ml-node+s472066n3995452...@n3.nabble.com> wrote: > Another thing you may wish to ponder is this blog entry from Mike > McCandless: > http://blog.mikemccandless.com/2011/04/just-say-no-to-swapping.html > > In it, he discusses the poor interaction between OS swapping, and > long-neglected allocations in a JVM. You're on Linux, which has decent > control over swapping decisions, so you may find that a tweak is in order, > especially if you can discover evidence that the hard drive is being > worked hard during GC. If the problem exists, it might be especially > pronounced in your large JVM. > > I have no direct evidence of thrashing during GC (I am not sure how to go > about gathering such evidence), but I have seen, on a Windows machine, a > Tomcat running Solr refuse to shut down for many minutes, while a Resource > Monitor session reports that that same Tomcat process is frantically > reading from the page file the whole time. So there is something besides > plausibility to the idea. > > -- Bryan > > > -----Original Message----- > > From: Mou [mailto:[hidden > > email]<http://user/SendEmail.jtp?type=node&node=3995452&i=0>] > > > Sent: Monday, July 16, 2012 9:09 PM > > To: [hidden email]<http://user/SendEmail.jtp?type=node&node=3995452&i=1> > > Subject: Re: Using Solr 3.4 running on tomcat7 - very slow search > > > > Thanks Brian. Excellent suggestion. > > > > I haven't used VisualVM before but I am going to use it to see where CPU > > is > > going. I saw that CPU is overly used. I haven't seen so much CPU use in > > testing. > > Although I think GC is not a problem, splitting the jvm per shard would > be > > a good idea. > > > > > > On Mon, Jul 16, 2012 at 9:44 PM, Bryan Loofbourrow [via Lucene] < > > [hidden email] <http://user/SendEmail.jtp?type=node&node=3995452&i=2>> > wrote: > > > > > 5 min is ridiculously long for a query that used to take 65ms. That > > ought > > > to be a great clue. The only two things I've seen that could cause > that > > > are thrashing, or GC. Hard to see how it could be thrashing, given > your > > > hardware, so I'd initially suspect GC. > > > > > > Aim VisualVM at the JVM. It shows how much CPU goes to GC over time, > in > > > a > > > nice blue line. And if it's not GC, try out its Sampler tab, and see > > where > > > the CPU is spending its time. > > > > > > FWIW, when asked at what point one would want to split JVMs and shard, > > on > > > the same machine, Grant Ingersoll mentioned 16GB, and precisely for GC > > > cost reasons. You're way above that. Maybe multiple JVMs and sharding, > > > even on the same machine, would serve you better than a monster 70GB > > JVM. > > > > > > -- Bryan > > > > > > > -----Original Message----- > > > > From: Mou [mailto:[hidden > > email]<http://user/SendEmail.jtp?type=node&node=3995446&i=0>] > > > > > > > Sent: Monday, July 16, 2012 7:43 PM > > > > To: [hidden > > email]<http://user/SendEmail.jtp?type=node&node=3995446&i=1> > > > > Subject: Using Solr 3.4 running on tomcat7 - very slow search > > > > > > > > Hi, > > > > > > > > Our index is divided into two shards and each of them has 120M docs > , > > > > total > > > > size 75G in each core. > > > > The server is a pretty good one , jvm is given memory of 70G and > about > > > > same > > > > is left for OS (SLES 11) . > > > > > > > > We use all dynamic fields except th eunique id and are using long > > > queries > > > > but almost all of them are filter queires, Each query may have 10 > -30 > > fq > > > > parameters. > > > > > > > > When I tested the index ( same size) but with max heap size 40 G, > > > queries > > > > > > > were blazing fast. I used solrmeter to load test and it was happily > > > > serving > > > > 12000 queries or more per min with avg 65 ms qtime.We had an > excellent > > > > filtercache hit ratio. > > > > > > > > This index is only used for searching and being replicated every 7 > sec > > > > > from > > > > the master. > > > > > > > > But now in production server it is horribly slow and taking 5 > > > mins(qtime) > > > > > > > to > > > > return a query ( same query). > > > > What could go wrong? > > > > > > > > Really appreciate your suggestions on debugging this thing.. > > > > > > > > > > > > > > > > -- > > > > View this message in context: > > http://lucene.472066.n3.nabble.com/Using- > > > > Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436.html > > > > Sent from the Solr - User mailing list archive at Nabble.com. > > > > > > > > > ------------------------------ > > > If you reply to this email, your message will be added to the > > discussion > > > below: > > > > > > http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7- > > very-slow-search-tp3995436p3995446.html > > > To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow > > search, click > > > > > > here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=uns > > > ubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg > > 1MTA5MTUw> > > > . > > > > > > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=mac > > > ro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespace > > s.BasicNamespace-nabble.view.web.template.NabbleNamespace- > > > nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21na > > bble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml- > > send_instant_email%21nabble%3Aemail.naml> > > > > > > > > > -- > > View this message in context: http://lucene.472066.n3.nabble.com/Using- > > Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995449.html > > Sent from the Solr - User mailing list archive at Nabble.com. > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995452.html > To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow search, > click > here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg1MTA5MTUw> > . > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995631.html Sent from the Solr - User mailing list archive at Nabble.com.
