Yonik, I got yourkit setup to profile the Tomcat instance and as you will see in the graph below all of the http threads are blocked (red) until around 4:40. This is the point where the instance becomes responsive and CPU usage drops. I have also ruled out GC being the issue by using the GC monitoring in yourkit. Let me know your thoughts and if you have any questions.
Thanks for your assistance. Thanks, John -- John Williams System Administrator 37signals
<<inline: Screen shot 2010-03-09 at 10.35.15 AM.png>>
On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: > On Mon, Mar 8, 2010 at 6:07 PM, John Williams <j...@37signals.com> wrote: >> Yonik, >> >> In all cases our "autowarmCount" is set to 0. Also, here is a link to our >> config. http://pastebin.com/iUgruqPd > > Weird... on a quick glance, I don't see anything in your config that > would cause work to be done on a commit. > I expected something like autowarming, or rebuilding a spellcheck > index, etc. I assume this is happening even w/o any requests hitting > the server? > > Could it be GC? You could use -verbose:gc or jconsole to check if > this corresponds to a big GC (which could naturally hit on an index > change). 5 minutes is really excessive though, and I wouldn't expect > it on startup. > > If it's not GC, perhaps the next step is to get some stack traces > during the spike (or use a profiler) to figure out where the time is > being spent. And verify that the solrconfig.xml shown actually still > matches the one you provided. > > -Yonik > http://www.lucidimagination.com > > > >> Thanks, >> John >> >> -- >> John Williams >> System Administrator >> 37signals >> >> On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: >> >>> Is this just autowarming? >>> Check your autowarmCount parameters in solrconfig.xml >>> >>> -Yonik >>> http://www.lucidimagination.com >>> >>> On Mon, Mar 8, 2010 at 5:37 PM, John Williams <j...@37signals.com> wrote: >>>> Good afternoon. >>>> >>>> We have been experiencing an odd issue with one of our Solr nodes. Upon >>>> startup or when bringing in a new index we get a CPU spike for 5 minutes >>>> or so. I have attached a graph of this spike. During this time simple >>>> queries return without a problem but more complex queries do not return. >>>> Here are some more details about the instance: >>>> >>>> Index Size: ~16G >>>> Max Heap: 6144M >>>> GC Option: -XX:+UseConcMarkSweepGC >>>> System Memory: 16G >>>> >>>> We have a very similar instance to this one but with a much larger index >>>> that we are not seeing this sort of issue. >>>> >>>> Your help is greatly appreciated. Let me know if you need any additional >>>> information. >>>> >>>> Thanks, >>>> John >>>> >>>> -- >>>> John Williams >>>> System Administrator >>>> 37signals >>>> >>>> >>>> >> >>
smime.p7s
Description: S/MIME cryptographic signature
