I'm going to defer to the folks who actually know the guts here. If you've turned down the cache entries for your Solr caches, you're pretty much left with Lucene caching which is a mystery...
Best Erick On Mon, Dec 5, 2011 at 9:23 AM, Jeff Crump <jeffrey.cr...@gmail.com> wrote: > Yes, and without doing much in the way of queries, either. Basically, our > test data has large numbers of distinct terms, each of which can be large > in themselves. Heap usage is a straight line -- up -- 75 percent of the > heap is consumed with byte[] allocations at the leaf of an object graph > like so: > > SolrCore > SolrIndexSearcher > DirectoryReader > SegmentReader > SegmentCoreReaders > PerFieldPostingsFormat$FieldsReader > ... > FST > byte[] > > Our application is less concerned with query performance than it is with > making sure our index doesn't OOM. My suspicion is that we're looking at > just in-memory representation of the index rather than any caching (it's > all turned down to levels suggested in other documentation); plus, we're > not doing much querying in this test anyway. > > Any suggestions or places to go for further information? > > On 5 December 2011 08:38, Erick Erickson <erickerick...@gmail.com> wrote: > >> There's no good way to say to Solr "Use only this >> much memory for searching". You can certainly >> limit the size somewhat by configuring your caches >> to be small. But if you're sorting, then Lucene will >> use up some cache space etc. >> >> Are you actually running into problems? >> >> Best >> Erick >> >> On Fri, Dec 2, 2011 at 2:26 PM, Jeff Crump <jeffrey.cr...@gmail.com> >> wrote: >> > Can anyone advise techniques for limiting the size of the RAM buffers to >> > begin with? As the index grows, I shouldn't have to keep increasing the >> > heap. We have a high-ingest, low-query-rate environment and I'm not as >> > much concerned with the query-time caches as I am with the segment core >> > readers/SolrIndexSearchers themselves. >> > >> > On 9 November 2011 06:10, Andre Bois-Crettez <andre.b...@kelkoo.com> >> wrote: >> > >> >> How much memory you actually allocate to the JVM ? >> >> http://wiki.apache.org/solr/**SolrPerformanceFactors#Memory_** >> >> allocated_to_the_Java_VM< >> http://wiki.apache.org/solr/SolrPerformanceFactors#Memory_allocated_to_the_Java_VM >> > >> >> You need to increase the -Xmx value, otherwise your large ram buffers >> >> won't fit in the java heap. >> >> >> >> >> >> >> >> sivaprasad wrote: >> >> >> >>> Hi, >> >>> >> >>> I am getting the following error during the indexing.I am trying to >> index >> >>> 14 >> >>> million records but the document size is very minimal. >> >>> >> >>> *Error:* >> >>> 2011-11-08 14:53:24,634 ERROR [STDERR] (Thread-12) >> >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >>> >> >>> >> >>> >> >> [...] >> >> >> >> Do i need to increase the heap size for JVM? >> >>> >> >>> My solrconfig settings are given below. >> >>> >> >>> <indexDefaults> >> >>> <useCompoundFile>false</**useCompoundFile> >> >>> >> >>> <mergeFactor>25</mergeFactor> >> >>> <maxBufferedDocs>2</**maxBufferedDocs> >> >>> <ramBufferSizeMB>1024</**ramBufferSizeMB> >> >>> <maxMergeDocs>2147483647</maxMergeDocs> >> >>> <maxFieldLength>10000</**maxFieldLength> >> >>> <writeLockTimeout>1000</**writeLockTimeout> >> >>> <commitLockTimeout>10000</**commitLockTimeout> >> >>> >> >>> and the main index values are >> >>> <useCompoundFile>false</**useCompoundFile> >> >>> <ramBufferSizeMB>512</**ramBufferSizeMB> >> >>> <mergeFactor>10</mergeFactor> >> >>> <maxMergeDocs>2147483647</maxMergeDocs> >> >>> <maxFieldLength>10000</**maxFieldLength> >> >>> >> >>> Do i need to increase the ramBufferSizeMB to a little higher? >> >>> >> >>> Please provide your inputs. >> >>> >> >>> Regards, >> >>> Siva >> >>> >> >>> -- >> >>> View this message in context: http://lucene.472066.n3.** >> >>> >> nabble.com/Out-of-memory-**during-the-indexing-**tp3492701p3492701.html< >> http://lucene.472066.n3.nabble.com/Out-of-memory-during-the-indexing-tp3492701p3492701.html >> > >> >>> Sent from the Solr - User mailing list archive at Nabble.com. >> >>> >> >>> >> >>> >> >> >> >> -- >> >> André Bois-Crettez >> >> >> >> Search technology, Kelkoo >> >> http://www.kelkoo.com/ >> >> >> >> >>
