Re: Solr Startup CPU Spike
Yonik, We are on Solr 1.3. The total number of documents is 54173459. Let me know if need any additional info. Thanks, John -- John Williams System Administrator 37signals On Mar 9, 2010, at 11:39 AM, Yonik Seeley wrote: > Ahhh, FieldCache loading... what version of Solr are you using? > It's interesting it would take that long to load too (and maxing out > one CPU - doesn't look particularly IO bound). How many documents are > in this index? > > -Yonik > > > On Tue, Mar 9, 2010 at 12:33 PM, John Williams wrote: >> Yonik, >> >> I have provided an image below gives details on what is causing the blocked >> http thread. Is there any way to resolve this issue. >> >> Thanks, >> John >> >> -- >> John Williams >> System Administrator >> 37signals >> >> >> >> On Mar 9, 2010, at 10:41 AM, John Williams wrote: >> >>> Yonik, >>> >>> I got yourkit setup to profile the Tomcat instance and as you will see in >>> the graph below all of the http threads are blocked (red) until around >>> 4:40. This is the point where the instance becomes responsive and CPU usage >>> drops. I have also ruled out GC being the issue by using the GC monitoring >>> in yourkit. Let me know your thoughts and if you have any questions. >>> >>> Thanks for your assistance. >>> >>> Thanks, >>> John >>> >>> -- >>> John Williams >>> System Administrator >>> 37signals >>> >>> >>> On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: >>> On Mon, Mar 8, 2010 at 6:07 PM, John Williams wrote: > Yonik, > > In all cases our "autowarmCount" is set to 0. Also, here is a link to our > config. http://pastebin.com/iUgruqPd Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com > Thanks, > John > > -- > John Williams > System Administrator > 37signals > > On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: > >> Is this just autowarming? >> Check your autowarmCount parameters in solrconfig.xml >> >> -Yonik >> http://www.lucidimagination.com >> >> On Mon, Mar 8, 2010 at 5:37 PM, John Williams wrote: >>> Good afternoon. >>> >>> We have been experiencing an odd issue with one of our Solr nodes. Upon >>> startup or when bringing in a new index we get a CPU spike for 5 >>> minutes or so. I have attached a graph of this spike. During this time >>> simple queries return without a problem but more complex queries do not >>> return. Here are some more details about the instance: >>> >>> Index Size: ~16G >>> Max Heap: 6144M >>> GC Option: -XX:+UseConcMarkSweepGC >>> System Memory: 16G >>> >>> We have a very similar instance to this one but with a much larger >>> index that we are not seeing this sort of issue. >>> >>> Your help is greatly appreciated. Let me know if you need any >>> additional information. >>> >>> Thanks, >>> John >>> >>> -- >>> John Williams >>> System Administrator >>> 37signals >>> >>> >>> > > >>> >> >> >> smime.p7s Description: S/MIME cryptographic signature
Re: Solr Startup CPU Spike
Mark, I am trying to load that url but its taking quite a while. I will let you know if/when it loads. -John -- John Williams System Administrator 37signals On Mar 9, 2010, at 11:38 AM, Mark Miller wrote: > Ah - loading the fieldcache - do you have a *lot* of unique terms in the > fields you are sorting/faceting on? > > localhost:8983/solr/admin/luke is helpful for checking this. > > > -- > - Mark > > http://www.lucidimagination.com > > > > On 03/09/2010 12:33 PM, John Williams wrote: >> Yonik, >> >> I have provided an image below gives details on what is causing the blocked >> http thread. Is there any way to resolve this issue. >> >> Thanks, >> John >> >> -- >> John Williams >> System Administrator >> 37signals >> >> >> >> >> On Mar 9, 2010, at 10:41 AM, John Williams wrote: >> >> >>> Yonik, >>> >>> I got yourkit setup to profile the Tomcat instance and as you will see in >>> the graph below all of the http threads are blocked (red) until around >>> 4:40. This is the point where the instance becomes responsive and CPU usage >>> drops. I have also ruled out GC being the issue by using the GC monitoring >>> in yourkit. Let me know your thoughts and if you have any questions. >>> >>> Thanks for your assistance. >>> >>> Thanks, >>> John >>> >>> -- >>> John Williams >>> System Administrator >>> 37signals >>> >>> >>> On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: >>> >>> On Mon, Mar 8, 2010 at 6:07 PM, John Williams wrote: > Yonik, > > In all cases our "autowarmCount" is set to 0. Also, here is a link to our > config. http://pastebin.com/iUgruqPd > Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com > Thanks, > John > > -- > John Williams > System Administrator > 37signals > > On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: > > >> Is this just autowarming? >> Check your autowarmCount parameters in solrconfig.xml >> >> -Yonik >> http://www.lucidimagination.com >> >> On Mon, Mar 8, 2010 at 5:37 PM, John Williams wrote: >> >>> Good afternoon. >>> >>> We have been experiencing an odd issue with one of our Solr nodes. Upon >>> startup or when bringing in a new index we get a CPU spike for 5 >>> minutes or so. I have attached a graph of this spike. During this time >>> simple queries return without a problem but more complex queries do not >>> return. Here are some more details about the instance: >>> >>> Index Size: ~16G >>> Max Heap: 6144M >>> GC Option: -XX:+UseConcMarkSweepGC >>> System Memory: 16G >>> >>> We have a very similar instance to this one but with a much larger >>> index that we are not seeing this sort of issue. >>> >>> Your help is greatly appreciated. Let me know if you need any >>> additional information. >>> >>> Thanks, >>> John >>> >>> -- >>> John Williams >>> System Administrator >>> 37signals >>> >>> >>> >>> > > >>> >> > > > smime.p7s Description: S/MIME cryptographic signature
Re: Solr Startup CPU Spike
Ahhh, FieldCache loading... what version of Solr are you using? It's interesting it would take that long to load too (and maxing out one CPU - doesn't look particularly IO bound). How many documents are in this index? -Yonik On Tue, Mar 9, 2010 at 12:33 PM, John Williams wrote: > Yonik, > > I have provided an image below gives details on what is causing the blocked > http thread. Is there any way to resolve this issue. > > Thanks, > John > > -- > John Williams > System Administrator > 37signals > > > > On Mar 9, 2010, at 10:41 AM, John Williams wrote: > >> Yonik, >> >> I got yourkit setup to profile the Tomcat instance and as you will see in >> the graph below all of the http threads are blocked (red) until around >> 4:40. This is the point where the instance becomes responsive and CPU usage >> drops. I have also ruled out GC being the issue by using the GC monitoring >> in yourkit. Let me know your thoughts and if you have any questions. >> >> Thanks for your assistance. >> >> Thanks, >> John >> >> -- >> John Williams >> System Administrator >> 37signals >> >> >> On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: >> >>> On Mon, Mar 8, 2010 at 6:07 PM, John Williams wrote: Yonik, In all cases our "autowarmCount" is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd >>> >>> Weird... on a quick glance, I don't see anything in your config that >>> would cause work to be done on a commit. >>> I expected something like autowarming, or rebuilding a spellcheck >>> index, etc. I assume this is happening even w/o any requests hitting >>> the server? >>> >>> Could it be GC? You could use -verbose:gc or jconsole to check if >>> this corresponds to a big GC (which could naturally hit on an index >>> change). 5 minutes is really excessive though, and I wouldn't expect >>> it on startup. >>> >>> If it's not GC, perhaps the next step is to get some stack traces >>> during the spike (or use a profiler) to figure out where the time is >>> being spent. And verify that the solrconfig.xml shown actually still >>> matches the one you provided. >>> >>> -Yonik >>> http://www.lucidimagination.com >>> >>> >>> Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: > Is this just autowarming? > Check your autowarmCount parameters in solrconfig.xml > > -Yonik > http://www.lucidimagination.com > > On Mon, Mar 8, 2010 at 5:37 PM, John Williams wrote: >> Good afternoon. >> >> We have been experiencing an odd issue with one of our Solr nodes. Upon >> startup or when bringing in a new index we get a CPU spike for 5 minutes >> or so. I have attached a graph of this spike. During this time simple >> queries return without a problem but more complex queries do not return. >> Here are some more details about the instance: >> >> Index Size: ~16G >> Max Heap: 6144M >> GC Option: -XX:+UseConcMarkSweepGC >> System Memory: 16G >> >> We have a very similar instance to this one but with a much larger index >> that we are not seeing this sort of issue. >> >> Your help is greatly appreciated. Let me know if you need any additional >> information. >> >> Thanks, >> John >> >> -- >> John Williams >> System Administrator >> 37signals >> >> >> >> > > >
Re: Solr Startup CPU Spike
Ah - loading the fieldcache - do you have a *lot* of unique terms in the fields you are sorting/faceting on? localhost:8983/solr/admin/luke is helpful for checking this. -- - Mark http://www.lucidimagination.com On 03/09/2010 12:33 PM, John Williams wrote: Yonik, I have provided an image below gives details on what is causing the blocked http thread. Is there any way to resolve this issue. Thanks, John -- John Williams System Administrator 37signals On Mar 9, 2010, at 10:41 AM, John Williams wrote: Yonik, I got yourkit setup to profile the Tomcat instance and as you will see in the graph below all of the http threads are blocked (red) until around 4:40. This is the point where the instance becomes responsive and CPU usage drops. I have also ruled out GC being the issue by using the GC monitoring in yourkit. Let me know your thoughts and if you have any questions. Thanks for your assistance. Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: On Mon, Mar 8, 2010 at 6:07 PM, John Williams wrote: Yonik, In all cases our "autowarmCount" is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williams wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals
Re: Solr Startup CPU Spike
Yonik, I have provided an image below gives details on what is causing the blocked http thread. Is there any way to resolve this issue. Thanks, John -- John Williams System Administrator 37signals <> On Mar 9, 2010, at 10:41 AM, John Williams wrote: > Yonik, > > I got yourkit setup to profile the Tomcat instance and as you will see in the > graph below all of the http threads are blocked (red) until around 4:40. > This is the point where the instance becomes responsive and CPU usage drops. > I have also ruled out GC being the issue by using the GC monitoring in > yourkit. Let me know your thoughts and if you have any questions. > > Thanks for your assistance. > > Thanks, > John > > -- > John Williams > System Administrator > 37signals > > > On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: > >> On Mon, Mar 8, 2010 at 6:07 PM, John Williams wrote: >>> Yonik, >>> >>> In all cases our "autowarmCount" is set to 0. Also, here is a link to our >>> config. http://pastebin.com/iUgruqPd >> >> Weird... on a quick glance, I don't see anything in your config that >> would cause work to be done on a commit. >> I expected something like autowarming, or rebuilding a spellcheck >> index, etc. I assume this is happening even w/o any requests hitting >> the server? >> >> Could it be GC? You could use -verbose:gc or jconsole to check if >> this corresponds to a big GC (which could naturally hit on an index >> change). 5 minutes is really excessive though, and I wouldn't expect >> it on startup. >> >> If it's not GC, perhaps the next step is to get some stack traces >> during the spike (or use a profiler) to figure out where the time is >> being spent. And verify that the solrconfig.xml shown actually still >> matches the one you provided. >> >> -Yonik >> http://www.lucidimagination.com >> >> >> >>> Thanks, >>> John >>> >>> -- >>> John Williams >>> System Administrator >>> 37signals >>> >>> On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: >>> Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williams wrote: > Good afternoon. > > We have been experiencing an odd issue with one of our Solr nodes. Upon > startup or when bringing in a new index we get a CPU spike for 5 minutes > or so. I have attached a graph of this spike. During this time simple > queries return without a problem but more complex queries do not return. > Here are some more details about the instance: > > Index Size: ~16G > Max Heap: 6144M > GC Option: -XX:+UseConcMarkSweepGC > System Memory: 16G > > We have a very similar instance to this one but with a much larger index > that we are not seeing this sort of issue. > > Your help is greatly appreciated. Let me know if you need any additional > information. > > Thanks, > John > > -- > John Williams > System Administrator > 37signals > > > >>> >>> > smime.p7s Description: S/MIME cryptographic signature
Re: Solr Startup CPU Spike
Yonik, I got yourkit setup to profile the Tomcat instance and as you will see in the graph below all of the http threads are blocked (red) until around 4:40. This is the point where the instance becomes responsive and CPU usage drops. I have also ruled out GC being the issue by using the GC monitoring in yourkit. Let me know your thoughts and if you have any questions. Thanks for your assistance. Thanks, John -- John Williams System Administrator 37signals <> On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: > On Mon, Mar 8, 2010 at 6:07 PM, John Williams wrote: >> Yonik, >> >> In all cases our "autowarmCount" is set to 0. Also, here is a link to our >> config. http://pastebin.com/iUgruqPd > > Weird... on a quick glance, I don't see anything in your config that > would cause work to be done on a commit. > I expected something like autowarming, or rebuilding a spellcheck > index, etc. I assume this is happening even w/o any requests hitting > the server? > > Could it be GC? You could use -verbose:gc or jconsole to check if > this corresponds to a big GC (which could naturally hit on an index > change). 5 minutes is really excessive though, and I wouldn't expect > it on startup. > > If it's not GC, perhaps the next step is to get some stack traces > during the spike (or use a profiler) to figure out where the time is > being spent. And verify that the solrconfig.xml shown actually still > matches the one you provided. > > -Yonik > http://www.lucidimagination.com > > > >> Thanks, >> John >> >> -- >> John Williams >> System Administrator >> 37signals >> >> On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: >> >>> Is this just autowarming? >>> Check your autowarmCount parameters in solrconfig.xml >>> >>> -Yonik >>> http://www.lucidimagination.com >>> >>> On Mon, Mar 8, 2010 at 5:37 PM, John Williams wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals >> >> smime.p7s Description: S/MIME cryptographic signature
Re: Solr Startup CPU Spike
On Mon, Mar 8, 2010 at 6:07 PM, John Williams wrote: > Yonik, > > In all cases our "autowarmCount" is set to 0. Also, here is a link to our > config. http://pastebin.com/iUgruqPd Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com > Thanks, > John > > -- > John Williams > System Administrator > 37signals > > On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: > >> Is this just autowarming? >> Check your autowarmCount parameters in solrconfig.xml >> >> -Yonik >> http://www.lucidimagination.com >> >> On Mon, Mar 8, 2010 at 5:37 PM, John Williams wrote: >>> Good afternoon. >>> >>> We have been experiencing an odd issue with one of our Solr nodes. Upon >>> startup or when bringing in a new index we get a CPU spike for 5 minutes or >>> so. I have attached a graph of this spike. During this time simple queries >>> return without a problem but more complex queries do not return. Here are >>> some more details about the instance: >>> >>> Index Size: ~16G >>> Max Heap: 6144M >>> GC Option: -XX:+UseConcMarkSweepGC >>> System Memory: 16G >>> >>> We have a very similar instance to this one but with a much larger index >>> that we are not seeing this sort of issue. >>> >>> Your help is greatly appreciated. Let me know if you need any additional >>> information. >>> >>> Thanks, >>> John >>> >>> -- >>> John Williams >>> System Administrator >>> 37signals >>> >>> >>> > >
Re: Solr Startup CPU Spike
Yonik, In all cases our "autowarmCount" is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: > Is this just autowarming? > Check your autowarmCount parameters in solrconfig.xml > > -Yonik > http://www.lucidimagination.com > > On Mon, Mar 8, 2010 at 5:37 PM, John Williams wrote: >> Good afternoon. >> >> We have been experiencing an odd issue with one of our Solr nodes. Upon >> startup or when bringing in a new index we get a CPU spike for 5 minutes or >> so. I have attached a graph of this spike. During this time simple queries >> return without a problem but more complex queries do not return. Here are >> some more details about the instance: >> >> Index Size: ~16G >> Max Heap: 6144M >> GC Option: -XX:+UseConcMarkSweepGC >> System Memory: 16G >> >> We have a very similar instance to this one but with a much larger index >> that we are not seeing this sort of issue. >> >> Your help is greatly appreciated. Let me know if you need any additional >> information. >> >> Thanks, >> John >> >> -- >> John Williams >> System Administrator >> 37signals >> >> >> smime.p7s Description: S/MIME cryptographic signature
Re: Solr Startup CPU Spike
Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williams wrote: > Good afternoon. > > We have been experiencing an odd issue with one of our Solr nodes. Upon > startup or when bringing in a new index we get a CPU spike for 5 minutes or > so. I have attached a graph of this spike. During this time simple queries > return without a problem but more complex queries do not return. Here are > some more details about the instance: > > Index Size: ~16G > Max Heap: 6144M > GC Option: -XX:+UseConcMarkSweepGC > System Memory: 16G > > We have a very similar instance to this one but with a much larger index that > we are not seeing this sort of issue. > > Your help is greatly appreciated. Let me know if you need any additional > information. > > Thanks, > John > > -- > John Williams > System Administrator > 37signals > > >
Solr Startup CPU Spike
Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals <> smime.p7s Description: S/MIME cryptographic signature
