RE: Java OutOfmemory error during autowarming
: Date: Mon, 1 Jun 2009 11:34:08 -0700 : From: Francis Yakin : Subject: RE: Java OutOfmemory error during autowarming ... : When it is initialized for the first time, how can I change it? Frnacis, I'm not sure if you already figured this out, but the point of hte documentation that Chris refered you to is that it is a completley under the covers caching mechanism of Lucene on a per field basis -- there is no way to change it, or configure it. It's sized according to the number of documents in the index and the number of terms in the Field. If you execute an operation that requires Lucene to use that datastructure for a field, then Lucene will build it, and cache it for as long as the IndexReader is open. : -Original Message- : From: Chris Harris [mailto:rygu...@gmail.com] : Sent: Sunday, May 31, 2009 3:00 PM : To: solr-user@lucene.apache.org : Subject: Re: Java OutOfmemory error during autowarming : : Solr offers no configuration for FieldCache, neither in solrconfig.xml nor anywhere else; rather, that cache gets populated automatically in the depths of Lucene when you do a sort (or also apparently, as Yonik says, when you use a field in a function query). : : From the wiki: 'Lucene has a low level FieldCache which is used for sorting (and in some cases faceting). This cache is not managed by Solr it has no configuration options and cannot be autowarmed -- it is initialized the first time it is used for each Searcher.' ( : http://wiki.apache.org/solr/SolrCaching) : : 2009/5/29 Francis Yakin fya...@liquid.com : : : I know, but the FieldCache is not in the solrconfig.xml : : : -Original Message- : From: Yonik Seeley [mailto:ysee...@gmail.com] : Sent: Friday, May 29, 2009 10:47 AM : To: solr-user@lucene.apache.org : Subject: Re: Java OutOfmemory error during autowarming : : On Fri, May 29, 2009 at 1:44 PM, Francis Yakin fya...@liquid.com wrote: : : There is no FieldCache entries in solrconfig.xml ( BTW we are : running version 1.2.0) : : Lucene FieldCache entries are created when you sort on a field or when : you use a field in a function query. : : -Yonik : : -Hoss
RE: Java OutOfmemory error during autowarming
Thanks Chris for the update. Right now we still having constant issue with our Production slaves. Do you think when I upgrade Solr to 1.3.0 this will fix the issue. I heard that in 1.3.0 there is parameter that you can set: queryResultMaxDocsCached200/queryResultMaxDocsCached Francis -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Friday, June 19, 2009 5:49 PM To: 'solr-user@lucene.apache.org' Subject: RE: Java OutOfmemory error during autowarming : Date: Mon, 1 Jun 2009 11:34:08 -0700 : From: Francis Yakin : Subject: RE: Java OutOfmemory error during autowarming ... : When it is initialized for the first time, how can I change it? Frnacis, I'm not sure if you already figured this out, but the point of hte documentation that Chris refered you to is that it is a completley under the covers caching mechanism of Lucene on a per field basis -- there is no way to change it, or configure it. It's sized according to the number of documents in the index and the number of terms in the Field. If you execute an operation that requires Lucene to use that datastructure for a field, then Lucene will build it, and cache it for as long as the IndexReader is open. : -Original Message- : From: Chris Harris [mailto:rygu...@gmail.com] : Sent: Sunday, May 31, 2009 3:00 PM : To: solr-user@lucene.apache.org : Subject: Re: Java OutOfmemory error during autowarming : : Solr offers no configuration for FieldCache, neither in solrconfig.xml nor anywhere else; rather, that cache gets populated automatically in the depths of Lucene when you do a sort (or also apparently, as Yonik says, when you use a field in a function query). : : From the wiki: 'Lucene has a low level FieldCache which is used for sorting (and in some cases faceting). This cache is not managed by Solr it has no configuration options and cannot be autowarmed -- it is initialized the first time it is used for each Searcher.' ( : http://wiki.apache.org/solr/SolrCaching) : : 2009/5/29 Francis Yakin fya...@liquid.com : : : I know, but the FieldCache is not in the solrconfig.xml : : : -Original Message- : From: Yonik Seeley [mailto:ysee...@gmail.com] : Sent: Friday, May 29, 2009 10:47 AM : To: solr-user@lucene.apache.org : Subject: Re: Java OutOfmemory error during autowarming : : On Fri, May 29, 2009 at 1:44 PM, Francis Yakin fya...@liquid.com wrote: : : There is no FieldCache entries in solrconfig.xml ( BTW we are : running version 1.2.0) : : Lucene FieldCache entries are created when you sort on a field or when : you use a field in a function query. : : -Yonik : : -Hoss
RE: Java OutOfmemory error during autowarming
Hi Chris, I am new in solr. When it is initialized for the first time, how can I change it? Thanks Francis -Original Message- From: Chris Harris [mailto:rygu...@gmail.com] Sent: Sunday, May 31, 2009 3:00 PM To: solr-user@lucene.apache.org Subject: Re: Java OutOfmemory error during autowarming Solr offers no configuration for FieldCache, neither in solrconfig.xml nor anywhere else; rather, that cache gets populated automatically in the depths of Lucene when you do a sort (or also apparently, as Yonik says, when you use a field in a function query). From the wiki: 'Lucene has a low level FieldCache which is used for sorting (and in some cases faceting). This cache is not managed by Solr it has no configuration options and cannot be autowarmed -- it is initialized the first time it is used for each Searcher.' ( http://wiki.apache.org/solr/SolrCaching) 2009/5/29 Francis Yakin fya...@liquid.com I know, but the FieldCache is not in the solrconfig.xml -Original Message- From: Yonik Seeley [mailto:ysee...@gmail.com] Sent: Friday, May 29, 2009 10:47 AM To: solr-user@lucene.apache.org Subject: Re: Java OutOfmemory error during autowarming On Fri, May 29, 2009 at 1:44 PM, Francis Yakin fya...@liquid.com wrote: There is no FieldCache entries in solrconfig.xml ( BTW we are running version 1.2.0) Lucene FieldCache entries are created when you sort on a field or when you use a field in a function query. -Yonik
Re: Java OutOfmemory error during autowarming
Solr offers no configuration for FieldCache, neither in solrconfig.xml nor anywhere else; rather, that cache gets populated automatically in the depths of Lucene when you do a sort (or also apparently, as Yonik says, when you use a field in a function query). From the wiki: 'Lucene has a low level FieldCache which is used for sorting (and in some cases faceting). This cache is not managed by Solr it has no configuration options and cannot be autowarmed -- it is initialized the first time it is used for each Searcher.' ( http://wiki.apache.org/solr/SolrCaching) 2009/5/29 Francis Yakin fya...@liquid.com I know, but the FieldCache is not in the solrconfig.xml -Original Message- From: Yonik Seeley [mailto:ysee...@gmail.com] Sent: Friday, May 29, 2009 10:47 AM To: solr-user@lucene.apache.org Subject: Re: Java OutOfmemory error during autowarming On Fri, May 29, 2009 at 1:44 PM, Francis Yakin fya...@liquid.com wrote: There is no FieldCache entries in solrconfig.xml ( BTW we are running version 1.2.0) Lucene FieldCache entries are created when you sort on a field or when you use a field in a function query. -Yonik
Re: Java OutOfmemory error during autowarming
It's probably not the size of the query cache, but the size of the FieldCache entries that are used for sorting and function queries (that's the only thing that should be allocating huge arrays like that). What fields do you sort on or use function queries on? There may be a way to decrease the memory consumption. -Yonik http://www.lucidimagination.com On Fri, May 29, 2009 at 1:02 AM, Francis Yakin fya...@liquid.com wrote: During auto-warming of solr search on QueryResultKey, Our Production solrslaves errors throw OutOfMemory error and application need to be bounced. Here is the error logs: SEVERE: Error during auto-warming of key:org.apache.solr.search.queryresult...@9acd5d67mailto:org.apache.solr.search.queryresult...@9acd5d67:java.lang.OutOfMemoryError: allocLargeObject OrArray - Object size: 25291544, Num elements: 6322881 SEVERE: Error during auto-warming of key:org.apache.solr.search.queryresult...@84554284mailto:org.apache.solr.search.queryresult...@84554284:java.lang.OutOfMemoryError: allocLargeObject OrArray - Object size: 579368, Num elements: 144837 SEVERE: Error during auto-warming of key:org.apache.solr.search.queryresult...@8c1e63e4mailto:org.apache.solr.search.queryresult...@8c1e63e4:java.lang.OutOfMemoryError: allocLargeObject OrArray - Object size: 4280976, Num elements: 1070240 I read some suggestions from google that increasing the size of queryResultCache in the solrconfig.xml file will help, here is the suggestion: This is not directly related to the jvm heap size. During slave replication the current Searcher is used as the source of auto-warming. When a new searcher is opened, its caches may be prepopulated or autowarmed using data from caches in the old searcher. It sounds like caching configuration problem to me as is evident from the error message Error during auto-warming of key. I suggest increasing the size of the query cache on one of the slave servers and monitor it over next week or so and then incrementally roll this change to rest of the slave boxes. This is the change that you need to make in solrconfig.xml Right now this what I have: queryResultCache class=solr.LRUCache size=512 initialSize=512 autowarmCount=256/ Should I increase size to 1024 will help? Any input? Thanks Francis
RE: Java OutOfmemory error during autowarming
There is no FieldCache entries in solrconfig.xml ( BTW we are running version 1.2.0) -Original Message- From: Yonik Seeley [mailto:ysee...@gmail.com] Sent: Friday, May 29, 2009 9:12 AM To: solr-user@lucene.apache.org Subject: Re: Java OutOfmemory error during autowarming It's probably not the size of the query cache, but the size of the FieldCache entries that are used for sorting and function queries (that's the only thing that should be allocating huge arrays like that). What fields do you sort on or use function queries on? There may be a way to decrease the memory consumption. -Yonik http://www.lucidimagination.com On Fri, May 29, 2009 at 1:02 AM, Francis Yakin fya...@liquid.com wrote: During auto-warming of solr search on QueryResultKey, Our Production solrslaves errors throw OutOfMemory error and application need to be bounced. Here is the error logs: SEVERE: Error during auto-warming of key:org.apache.solr.search.queryresult...@9acd5d67mailto:org.apache.s olr.search.queryresult...@9acd5d67:java.lang.OutOfMemoryError: allocLargeObject OrArray - Object size: 25291544, Num elements: 6322881 SEVERE: Error during auto-warming of key:org.apache.solr.search.queryresult...@84554284mailto:org.apache.s olr.search.queryresult...@84554284:java.lang.OutOfMemoryError: allocLargeObject OrArray - Object size: 579368, Num elements: 144837 SEVERE: Error during auto-warming of key:org.apache.solr.search.queryresult...@8c1e63e4mailto:org.apache.s olr.search.queryresult...@8c1e63e4:java.lang.OutOfMemoryError: allocLargeObject OrArray - Object size: 4280976, Num elements: 1070240 I read some suggestions from google that increasing the size of queryResultCache in the solrconfig.xml file will help, here is the suggestion: This is not directly related to the jvm heap size. During slave replication the current Searcher is used as the source of auto-warming. When a new searcher is opened, its caches may be prepopulated or autowarmed using data from caches in the old searcher. It sounds like caching configuration problem to me as is evident from the error message Error during auto-warming of key. I suggest increasing the size of the query cache on one of the slave servers and monitor it over next week or so and then incrementally roll this change to rest of the slave boxes. This is the change that you need to make in solrconfig.xml Right now this what I have: queryResultCache class=solr.LRUCache size=512 initialSize=512 autowarmCount=256/ Should I increase size to 1024 will help? Any input? Thanks Francis
Re: Java OutOfmemory error during autowarming
On Fri, May 29, 2009 at 1:44 PM, Francis Yakin fya...@liquid.com wrote: There is no FieldCache entries in solrconfig.xml ( BTW we are running version 1.2.0) Lucene FieldCache entries are created when you sort on a field or when you use a field in a function query. -Yonik
RE: Java OutOfmemory error during autowarming
I know, but the FieldCache is not in the solrconfig.xml -Original Message- From: Yonik Seeley [mailto:ysee...@gmail.com] Sent: Friday, May 29, 2009 10:47 AM To: solr-user@lucene.apache.org Subject: Re: Java OutOfmemory error during autowarming On Fri, May 29, 2009 at 1:44 PM, Francis Yakin fya...@liquid.com wrote: There is no FieldCache entries in solrconfig.xml ( BTW we are running version 1.2.0) Lucene FieldCache entries are created when you sort on a field or when you use a field in a function query. -Yonik
