Re: Solr 1.4 - stats page slow
FYI, I opened https://issues.apache.org/jira/browse/SOLR-2036 for this. -Yonik http://www.lucidimagination.com On Tue, Aug 10, 2010 at 8:35 PM, entdeveloper cameron.develo...@gmail.com wrote: Apologies if this was resolved, but we just deployed Solr 1.4.1 and the stats page takes over a minute to load for us as well and began causing OutOfMemory errors so we've had to refrain from hitting the page. From what I gather, it is the fieldCache part that's causing it. Was there ever an official fix or recommendation on how to disable the stats page from calculating the fieldCache entries? If we could just ignore it, I think we'd be good to go since I find this page very useful otherwise.
Re: Solr 1.4 - stats page slow
Apologies if this was resolved, but we just deployed Solr 1.4.1 and the stats page takes over a minute to load for us as well and began causing OutOfMemory errors so we've had to refrain from hitting the page. From what I gather, it is the fieldCache part that's causing it. Was there ever an official fix or recommendation on how to disable the stats page from calculating the fieldCache entries? If we could just ignore it, I think we'd be good to go since I find this page very useful otherwise. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-1-4-stats-page-slow-tp498810p1081193.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.4 - stats page slow
Yes, we do have some fields (like the creation date) that we use for both sorting and faceting. -Peter On Tue, Jan 26, 2010 at 8:55 PM, Yonik Seeley yo...@lucidimagination.com wrote: On Tue, Jan 26, 2010 at 8:49 PM, Peter Wolanin peter.wola...@acquia.com wrote: Sorry for not following up sooner- been a busy last couple weeks. We do see a significant instanity count - could this be due to updating indexes from the dev Solr build? E.g. on one server I see Do you both sort (or use a function query) and facet on the created field? Faceting on single-valued fields is still currently done at the top-level reader, while sorting and function queries are at a segment level. -Yonik http://www.lucidimagination.com -- Peter M. Wolanin, Ph.D. Momentum Specialist, Acquia. Inc. peter.wola...@acquia.com
Gathering metrics on 1.4 (was Re: Solr 1.4 - stats page slow)
Heya - So we just upgraded our Solr install to 1.4, and there's a great CPU drop and query response time drop. Good! But we're seeing the slowdown in the collection of statistics (stats.jsp) mentioned here: http://www.mail-archive.com/solr-user@lucene.apache.org/msg30224.html to the tune of taking 100 seconds just to load it, I suspect that it does indeed (as mentioned in the thread above) that is has something to do with the fieldCache additions in 1.4. Loading stats.jsp on 1.3 with the same index was pretty much instantaneous. We (probably obviously) take things like avgTimePerRequest and avgRequestsPerSecond, cache hit ratios, evictions, etc. from stats.jsp and use it for both alerting (on thresholds) as well as tracking/trending/graphing/etc. Question: is there any plan to return this page to its previous fastness? Either by making optional the fieldCache stuff, or having an alternate method to get those metrics? Because at the moment, running 1.4 in production means essentially flying blind without those. :) cheers, j
Re: Gathering metrics on 1.4 (was Re: Solr 1.4 - stats page slow)
john allspaw wrote: Heya - So we just upgraded our Solr install to 1.4, and there's a great CPU drop and query response time drop. Good! But we're seeing the slowdown in the collection of statistics (stats.jsp) mentioned here: http://www.mail-archive.com/solr-user@lucene.apache.org/msg30224.html to the tune of taking 100 seconds just to load it, I suspect that it does indeed (as mentioned in the thread above) that is has something to do with the fieldCache additions in 1.4. Loading stats.jsp on 1.3 with the same index was pretty much instantaneous. We (probably obviously) take things like avgTimePerRequest and avgRequestsPerSecond, cache hit ratios, evictions, etc. from stats.jsp and use it for both alerting (on thresholds) as well as tracking/trending/graphing/etc. Question: is there any plan to return this page to its previous fastness? Either by making optional the fieldCache stuff, or having an alternate method to get those metrics? Because at the moment, running 1.4 in production means essentially flying blind without those. :) cheers, j Yes, there is a plan for this. As far as I'm concerned, this is essentially a bug and it def needs to be fixed. Checking FieldCache size *needs* to be optional on the stat page. Not sure if there is a JIRA issue to fix this yet, but feel free to make one if there is not. Needs a resolution. There is also a new StatsRequestHandler in the works I believe. -- - Mark http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
Sorry for not following up sooner- been a busy last couple weeks. We do see a significant instanity count - could this be due to updating indexes from the dev Solr build? E.g. on one server I see stat name=insanity_count 61 /stat and entries like: stat name=insanity#0 SUBREADER: Found caches for decendents of org.apache.lucene.index.readonlydirectoryrea...@2b8d6cbf+created 'org.apache.lucene.index.readonlydirectoryrea...@2b8d6cbf'='created',class org.apache.lucene.search.FieldCache$StringIndex,null=org.apache.lucene.search.FieldCache$StringIndex#2002656056 (size =~ 74.4 KB) 'org.apache.lucene.store.niofsdirectory$niofsindexin...@47adeb94'='created',class org.apache.lucene.search.FieldCache$StringIndex,null=org.apache.lucene.search.FieldCache$StringIndex#1099177573 (size =~ 74.4 KB) /stat stat name=insanity#1 SUBREADER: Found caches for decendents of org.apache.lucene.index.readonlydirectoryrea...@d0340a9+created 'org.apache.lucene.index.readonlydirectoryrea...@d0340a9'='created',class org.apache.lucene.search.FieldCache$StringIndex,null=org.apache.lucene.search.FieldCache$StringIndex#868132357 (size =~ 831.2 KB) 'org.apache.lucene.store.niofsdirectory$niofsindexin...@78802615'='created',class org.apache.lucene.search.FieldCache$StringIndex,null=org.apache.lucene.search.FieldCache$StringIndex#1542727931 (size =~ 831.2 KB) /stat And I think it's higher on the one associated with the screenshot. using the lucene checkIndex tool does not show any errors. Most of what we want is returned by the Luke handler, except for the pending adds and deletes and the index size. I can hack around this by creating a greatly reduced stats.jsp, but I'd also liek to understand what we are experiencing. -Peter On Fri, Jan 8, 2010 at 1:38 PM, Mark Miller markrmil...@gmail.com wrote: Yonik Seeley wrote: On Fri, Jan 8, 2010 at 1:03 PM, Mark Miller markrmil...@gmail.com wrote: It should be fixed in trunk, but that was after 1.4. Currently, it should only do it if it sees insanity - which there shouldn't be any with stock Solr. http://svn.apache.org/viewvc/lucene/solr/tags/release-1.4.0/src/java/org/apache/solr/search/SolrFieldCacheMBean.java http://svn.apache.org/viewvc?view=revisionrevision=826788 Seems like it's there? Or was it a different commit? Perhaps there is just real instanity... which may be unavoidable at this point since not everything in solr is done per-segment yet. -Yonik http://www.lucidimagination.com Your right - when looking at the Solr release date, I quickly took the 10 as October - but it was 11/10, so it is in 1.4. So people seeing this should also being seeing an insanity count over one. I'd think that would be rarer than one this sounds like though ... whats left that could cause insanity? We should prob switch to never calculating the size unless an explicit param is pass to the stats page. -- - Mark http://www.lucidimagination.com -- Peter M. Wolanin, Ph.D. Momentum Specialist, Acquia. Inc. peter.wola...@acquia.com
Re: Solr 1.4 - stats page slow
On Tue, Jan 26, 2010 at 8:49 PM, Peter Wolanin peter.wola...@acquia.com wrote: Sorry for not following up sooner- been a busy last couple weeks. We do see a significant instanity count - could this be due to updating indexes from the dev Solr build? E.g. on one server I see Do you both sort (or use a function query) and facet on the created field? Faceting on single-valued fields is still currently done at the top-level reader, while sorting and function queries are at a segment level. -Yonik http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
Ah sorry - didn't realize attachments were stripped. Here's a web version: http://img.skitch.com/20100108-t99a1emmar32w9gkcfcius8afm.png -Peter On Thu, Jan 7, 2010 at 9:53 PM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: I'd love to see the screenshot, but it didn't come through - got stripped by ML manager. Maybe upload it somewhere? Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch - Original Message From: Peter Wolanin peter.wola...@acquia.com To: solr-user@lucene.apache.org Sent: Thu, January 7, 2010 9:32:26 PM Subject: Re: Solr 1.4 - stats page slow I recently noticed the same sort of thing. The attached screenshot shows the transition on a search server when we updated from a Solr 1.4 dev build (revision 779609 from 2009-05-28) to the Solr 1.4.0 released code. Every 3 hours we have a cron task to log some of the data from the stats.jsp page from each core (about 100 cores, most of which are small indexes). You can see there is a dramatic spiking of the load after the update - I think due to added reporting on that page such as from the lucene field cache. Is this amount of load expected? -Peter On Thu, Dec 24, 2009 at 12:23 PM, Jay Hill wrote: Also, what is your heap size and the amount of RAM on the machine? I've also noticed that, when watching memory usage through JConsole or YourKit while loading the stats page, the memory usage spikes dramatically - are you seeing this as well? -Jay On Thu, Dec 24, 2009 at 9:12 AM, Jay Hill wrote: I've noticed this as well, usually when working with a large field cache. I haven't done in-depth analysis of this yet, but it seems like when the stats page is trying to pull data from a large field cache it takes quite a long time. Are you doing a lot of sorting? If so, what are the field types of the fields you're sorting on? How large is the index both in document count and file size? Another approach to get data from the Solr instance would be to use JMX. And I've been working on a request handler (started by Erik Hatcher) that will provide the same information as the stats page, but a little more efficiently. I may try to put up a patch with this soon. -Jay On Wed, Dec 23, 2009 at 6:43 AM, Stephen Weiss wrote: We've been using Solr 1.4 for a few days now and one slight downside we've noticed is the stats page comes up very slowly for some reason - sometimes more than 10 seconds. We call this programmatically to retrieve the last commit date so that we can keep users from committing too frequently. This means some of our administration pages are now taking a long time to load. Is there anything we should be doing to ensure that this page comes up quickly? I see some notes on this back in October but it looks like that update should already be applied by now. Or, better yet, is there now a better way to just retrieve the last commit date from Solr without pulling all of the statistics? Thanks in advance. -- Steve -- Peter M. Wolanin, Ph.D. Momentum Specialist, Acquia. Inc. peter.wola...@acquia.com -- Peter M. Wolanin, Ph.D. Momentum Specialist, Acquia. Inc. peter.wola...@acquia.com
Re: Solr 1.4 - stats page slow
I thought this was fixed... http://issues.apache.org/jira/browse/SOLR-1292 http://www.lucidimagination.com/search/document/57103830f0655776/stats_page_slow_in_latest_nightly -Yonik http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
It's definitely still an issue. I've seen this with at least four different Solr implementations. It clearly seems to be a problem when there is a large field cache. It would be bad enough if the stats.jsp was just slow to load (usually takes 1 to 2 minutes), but when monitoring memory usage with jconsole there is a clear and serious spike as soon as the url for stats.jsp is hit, on occasions causing OutOfMemory Exceptions. -Jay On Fri, Jan 8, 2010 at 9:46 AM, Yonik Seeley yo...@lucidimagination.comwrote: I thought this was fixed... http://issues.apache.org/jira/browse/SOLR-1292 http://www.lucidimagination.com/search/document/57103830f0655776/stats_page_slow_in_latest_nightly -Yonik http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
Yonik Seeley wrote: I thought this was fixed... http://issues.apache.org/jira/browse/SOLR-1292 http://www.lucidimagination.com/search/document/57103830f0655776/stats_page_slow_in_latest_nightly -Yonik http://www.lucidimagination.com It should be fixed in trunk, but that was after 1.4. Currently, it should only do it if it sees insanity - which there shouldn't be any with stock Solr. -- - Mark http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
Mark Miller wrote: Yonik Seeley wrote: I thought this was fixed... http://issues.apache.org/jira/browse/SOLR-1292 http://www.lucidimagination.com/search/document/57103830f0655776/stats_page_slow_in_latest_nightly -Yonik http://www.lucidimagination.com It should be fixed in trunk, but that was after 1.4. Currently, it should only do it if it sees insanity - which there shouldn't be any with stock Solr. Though it would still be nice to have a url param that forced the size check no matter what. -- - Mark http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
On Fri, Jan 8, 2010 at 1:03 PM, Mark Miller markrmil...@gmail.com wrote: It should be fixed in trunk, but that was after 1.4. Currently, it should only do it if it sees insanity - which there shouldn't be any with stock Solr. http://svn.apache.org/viewvc/lucene/solr/tags/release-1.4.0/src/java/org/apache/solr/search/SolrFieldCacheMBean.java http://svn.apache.org/viewvc?view=revisionrevision=826788 Seems like it's there? Or was it a different commit? Perhaps there is just real instanity... which may be unavoidable at this point since not everything in solr is done per-segment yet. -Yonik http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
Yonik Seeley wrote: On Fri, Jan 8, 2010 at 1:03 PM, Mark Miller markrmil...@gmail.com wrote: It should be fixed in trunk, but that was after 1.4. Currently, it should only do it if it sees insanity - which there shouldn't be any with stock Solr. http://svn.apache.org/viewvc/lucene/solr/tags/release-1.4.0/src/java/org/apache/solr/search/SolrFieldCacheMBean.java http://svn.apache.org/viewvc?view=revisionrevision=826788 Seems like it's there? Or was it a different commit? Perhaps there is just real instanity... which may be unavoidable at this point since not everything in solr is done per-segment yet. -Yonik http://www.lucidimagination.com Your right - when looking at the Solr release date, I quickly took the 10 as October - but it was 11/10, so it is in 1.4. So people seeing this should also being seeing an insanity count over one. I'd think that would be rarer than one this sounds like though ... whats left that could cause insanity? We should prob switch to never calculating the size unless an explicit param is pass to the stats page. -- - Mark http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
: 2009-05-28) to the Solr 1.4.0 released code. Every 3 hours we have a : cron task to log some of the data from the stats.jsp page from each : core (about 100 cores, most of which are small indexes). 1) what stats are you actaully interested in? ... in Jay's case the LukeRequestHandler made more sense to get the data he wnated anyway. 2) what does the output of stats.jsp say when you see these load spikes? ... it should be fairly lightweight unless it detects some insanity in the way the FieldCaches are being used, in which case it does memory estimation to make it clear how significant the problem is. -Hoss
Re: Solr 1.4 - stats page slow
: We should prob switch to never calculating the size unless an explicit : param is pass to the stats page. I don't think that's as easy as it sounds considdering stats.jsp is just iterating over MBeans - that level of control over the Insanity checker is abstracted away. -Hoss
Re: Solr 1.4 - stats page slow
Chris Hostetter wrote: : We should prob switch to never calculating the size unless an explicit : param is pass to the stats page. I don't think that's as easy as it sounds considdering stats.jsp is just iterating over MBeans - that level of control over the Insanity checker is abstracted away. -Hoss With the current implementation - many other ways to do it though. I started on this method when I was originally working on this issue - don't think I still have the code though. -- - Mark http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
Actually my cases were all with customers I work with, not just one case. A common practice is to monitor cache stats to tune the caches properly. Also, noting the warmup times for new IndexSearchers, etc. I've worked with people that have excessive auto-warm count values which is causing extremely long warmup times for the new Searchers. So the stats.jsp page has always been a handy, simple tool to monitor this stuff and set caches appropriately. But at some point (around the release of 1.4) I started to notice this problem. Since it causes the memory spike it pretty much prevents the use of stats.jsp in production. I've had to resort to log-parsing and other tricks which is a bit of a waste since it was so simple to do before this surfaced. -Jay On Fri, Jan 8, 2010 at 10:41 AM, Chris Hostetter hossman_luc...@fucit.orgwrote: : 2009-05-28) to the Solr 1.4.0 released code. Every 3 hours we have a : cron task to log some of the data from the stats.jsp page from each : core (about 100 cores, most of which are small indexes). 1) what stats are you actaully interested in? ... in Jay's case the LukeRequestHandler made more sense to get the data he wnated anyway. 2) what does the output of stats.jsp say when you see these load spikes? ... it should be fairly lightweight unless it detects some insanity in the way the FieldCaches are being used, in which case it does memory estimation to make it clear how significant the problem is. -Hoss
Re: Solr 1.4 - stats page slow
Why don't we create a patch that allows one to check a box to get the Field Cache stats and then reload? Off by default. On Jan 8, 2010, at 2:33 PM, Jay Hill wrote: Actually my cases were all with customers I work with, not just one case. A common practice is to monitor cache stats to tune the caches properly. Also, noting the warmup times for new IndexSearchers, etc. I've worked with people that have excessive auto-warm count values which is causing extremely long warmup times for the new Searchers. So the stats.jsp page has always been a handy, simple tool to monitor this stuff and set caches appropriately. But at some point (around the release of 1.4) I started to notice this problem. Since it causes the memory spike it pretty much prevents the use of stats.jsp in production. I've had to resort to log-parsing and other tricks which is a bit of a waste since it was so simple to do before this surfaced. -Jay On Fri, Jan 8, 2010 at 10:41 AM, Chris Hostetter hossman_luc...@fucit.orgwrote: : 2009-05-28) to the Solr 1.4.0 released code. Every 3 hours we have a : cron task to log some of the data from the stats.jsp page from each : core (about 100 cores, most of which are small indexes). 1) what stats are you actaully interested in? ... in Jay's case the LukeRequestHandler made more sense to get the data he wnated anyway. 2) what does the output of stats.jsp say when you see these load spikes? ... it should be fairly lightweight unless it detects some insanity in the way the FieldCaches are being used, in which case it does memory estimation to make it clear how significant the problem is. -Hoss -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search
Re: Solr 1.4 - stats page slow
On Fri, Jan 8, 2010 at 1:38 PM, Mark Miller markrmil...@gmail.com wrote: Yonik Seeley wrote: So people seeing this should also being seeing an insanity count over one. I'd think that would be rarer than one this sounds like though ... whats left that could cause insanity? faceting on a field, ord(field), and stats component all use top-level fieldcache and will cause it in conjunction with sorting, other function queries, etc. -Yonik http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
Yonik Seeley wrote: ord(field) I thought you used those leaf readers to get around top level for ord? -- - Mark http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
On Fri, Jan 8, 2010 at 3:02 PM, Mark Miller markrmil...@gmail.com wrote: Yonik Seeley wrote: ord(field) I thought you used those leaf readers to get around top level for ord? ord()/rord() now works via top()... by popping back to the top level reader. -Yonik http://www.lucidimagination.com
Re: Solr 1.4 - stats page slow
I recently noticed the same sort of thing. The attached screenshot shows the transition on a search server when we updated from a Solr 1.4 dev build (revision 779609 from 2009-05-28) to the Solr 1.4.0 released code. Every 3 hours we have a cron task to log some of the data from the stats.jsp page from each core (about 100 cores, most of which are small indexes). You can see there is a dramatic spiking of the load after the update - I think due to added reporting on that page such as from the lucene field cache. Is this amount of load expected? -Peter On Thu, Dec 24, 2009 at 12:23 PM, Jay Hill jayallenh...@gmail.com wrote: Also, what is your heap size and the amount of RAM on the machine? I've also noticed that, when watching memory usage through JConsole or YourKit while loading the stats page, the memory usage spikes dramatically - are you seeing this as well? -Jay On Thu, Dec 24, 2009 at 9:12 AM, Jay Hill jayallenh...@gmail.com wrote: I've noticed this as well, usually when working with a large field cache. I haven't done in-depth analysis of this yet, but it seems like when the stats page is trying to pull data from a large field cache it takes quite a long time. Are you doing a lot of sorting? If so, what are the field types of the fields you're sorting on? How large is the index both in document count and file size? Another approach to get data from the Solr instance would be to use JMX. And I've been working on a request handler (started by Erik Hatcher) that will provide the same information as the stats page, but a little more efficiently. I may try to put up a patch with this soon. -Jay On Wed, Dec 23, 2009 at 6:43 AM, Stephen Weiss swe...@stylesight.comwrote: We've been using Solr 1.4 for a few days now and one slight downside we've noticed is the stats page comes up very slowly for some reason - sometimes more than 10 seconds. We call this programmatically to retrieve the last commit date so that we can keep users from committing too frequently. This means some of our administration pages are now taking a long time to load. Is there anything we should be doing to ensure that this page comes up quickly? I see some notes on this back in October but it looks like that update should already be applied by now. Or, better yet, is there now a better way to just retrieve the last commit date from Solr without pulling all of the statistics? Thanks in advance. -- Steve -- Peter M. Wolanin, Ph.D. Momentum Specialist, Acquia. Inc. peter.wola...@acquia.com
Re: Solr 1.4 - stats page slow
I'd love to see the screenshot, but it didn't come through - got stripped by ML manager. Maybe upload it somewhere? Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch - Original Message From: Peter Wolanin peter.wola...@acquia.com To: solr-user@lucene.apache.org Sent: Thu, January 7, 2010 9:32:26 PM Subject: Re: Solr 1.4 - stats page slow I recently noticed the same sort of thing. The attached screenshot shows the transition on a search server when we updated from a Solr 1.4 dev build (revision 779609 from 2009-05-28) to the Solr 1.4.0 released code. Every 3 hours we have a cron task to log some of the data from the stats.jsp page from each core (about 100 cores, most of which are small indexes). You can see there is a dramatic spiking of the load after the update - I think due to added reporting on that page such as from the lucene field cache. Is this amount of load expected? -Peter On Thu, Dec 24, 2009 at 12:23 PM, Jay Hill wrote: Also, what is your heap size and the amount of RAM on the machine? I've also noticed that, when watching memory usage through JConsole or YourKit while loading the stats page, the memory usage spikes dramatically - are you seeing this as well? -Jay On Thu, Dec 24, 2009 at 9:12 AM, Jay Hill wrote: I've noticed this as well, usually when working with a large field cache. I haven't done in-depth analysis of this yet, but it seems like when the stats page is trying to pull data from a large field cache it takes quite a long time. Are you doing a lot of sorting? If so, what are the field types of the fields you're sorting on? How large is the index both in document count and file size? Another approach to get data from the Solr instance would be to use JMX. And I've been working on a request handler (started by Erik Hatcher) that will provide the same information as the stats page, but a little more efficiently. I may try to put up a patch with this soon. -Jay On Wed, Dec 23, 2009 at 6:43 AM, Stephen Weiss wrote: We've been using Solr 1.4 for a few days now and one slight downside we've noticed is the stats page comes up very slowly for some reason - sometimes more than 10 seconds. We call this programmatically to retrieve the last commit date so that we can keep users from committing too frequently. This means some of our administration pages are now taking a long time to load. Is there anything we should be doing to ensure that this page comes up quickly? I see some notes on this back in October but it looks like that update should already be applied by now. Or, better yet, is there now a better way to just retrieve the last commit date from Solr without pulling all of the statistics? Thanks in advance. -- Steve -- Peter M. Wolanin, Ph.D. Momentum Specialist, Acquia. Inc. peter.wola...@acquia.com
Re: Solr 1.4 - stats page slow
Sorry, know I'm a little late in replying but the LukeRequestHandler tip was just what I needed! Thank you so much. -- Steve On Dec 25, 2009, at 2:03 AM, Chris Hostetter wrote: : I've noticed this as well, usually when working with a large field cache. I : haven't done in-depth analysis of this yet, but it seems like when the stats : page is trying to pull data from a large field cache it takes quite a long : time. In Solr 1.4, the stats page was modified to start reporting stats on the FieldCache (using the new FieldCache introspection API added by Lucene Java 2.9) so that may be what you are seeing. : more than 10 seconds. We call this programmatically to retrieve the last : commit date so that we can keep users from committing too frequently. This : means some of our administration pages are now taking a long time to load. i'm not really following this ... what piece of data from the stats.jsp are you using to compute/infer a commit date? if you are looking at registration date of the SolrIndexSearcher you can also get that from the LukeRequestHandler which is much more efficient (it has options for limiting the work it does)... http://localhost:8983/solr/admin/luke?numTerms=0fl=BOGUS -Hoss
Re: Solr 1.4 - stats page slow
I've noticed this as well, usually when working with a large field cache. I haven't done in-depth analysis of this yet, but it seems like when the stats page is trying to pull data from a large field cache it takes quite a long time. Are you doing a lot of sorting? If so, what are the field types of the fields you're sorting on? How large is the index both in document count and file size? Another approach to get data from the Solr instance would be to use JMX. And I've been working on a request handler (started by Erik Hatcher) that will provide the same information as the stats page, but a little more efficiently. I may try to put up a patch with this soon. -Jay On Wed, Dec 23, 2009 at 6:43 AM, Stephen Weiss swe...@stylesight.comwrote: We've been using Solr 1.4 for a few days now and one slight downside we've noticed is the stats page comes up very slowly for some reason - sometimes more than 10 seconds. We call this programmatically to retrieve the last commit date so that we can keep users from committing too frequently. This means some of our administration pages are now taking a long time to load. Is there anything we should be doing to ensure that this page comes up quickly? I see some notes on this back in October but it looks like that update should already be applied by now. Or, better yet, is there now a better way to just retrieve the last commit date from Solr without pulling all of the statistics? Thanks in advance. -- Steve
Re: Solr 1.4 - stats page slow
Also, what is your heap size and the amount of RAM on the machine? I've also noticed that, when watching memory usage through JConsole or YourKit while loading the stats page, the memory usage spikes dramatically - are you seeing this as well? -Jay On Thu, Dec 24, 2009 at 9:12 AM, Jay Hill jayallenh...@gmail.com wrote: I've noticed this as well, usually when working with a large field cache. I haven't done in-depth analysis of this yet, but it seems like when the stats page is trying to pull data from a large field cache it takes quite a long time. Are you doing a lot of sorting? If so, what are the field types of the fields you're sorting on? How large is the index both in document count and file size? Another approach to get data from the Solr instance would be to use JMX. And I've been working on a request handler (started by Erik Hatcher) that will provide the same information as the stats page, but a little more efficiently. I may try to put up a patch with this soon. -Jay On Wed, Dec 23, 2009 at 6:43 AM, Stephen Weiss swe...@stylesight.comwrote: We've been using Solr 1.4 for a few days now and one slight downside we've noticed is the stats page comes up very slowly for some reason - sometimes more than 10 seconds. We call this programmatically to retrieve the last commit date so that we can keep users from committing too frequently. This means some of our administration pages are now taking a long time to load. Is there anything we should be doing to ensure that this page comes up quickly? I see some notes on this back in October but it looks like that update should already be applied by now. Or, better yet, is there now a better way to just retrieve the last commit date from Solr without pulling all of the statistics? Thanks in advance. -- Steve
Re: Solr 1.4 - stats page slow
: I've noticed this as well, usually when working with a large field cache. I : haven't done in-depth analysis of this yet, but it seems like when the stats : page is trying to pull data from a large field cache it takes quite a long : time. In Solr 1.4, the stats page was modified to start reporting stats on the FieldCache (using the new FieldCache introspection API added by Lucene Java 2.9) so that may be what you are seeing. : more than 10 seconds. We call this programmatically to retrieve the last : commit date so that we can keep users from committing too frequently. This : means some of our administration pages are now taking a long time to load. i'm not really following this ... what piece of data from the stats.jsp are you using to compute/infer a commit date? if you are looking at registration date of the SolrIndexSearcher you can also get that from the LukeRequestHandler which is much more efficient (it has options for limiting the work it does)... http://localhost:8983/solr/admin/luke?numTerms=0fl=BOGUS -Hoss
