Hi,
Which database are you using? Can you send us data-config.xml?
What happens when you use default merge policy settings?
What happens when you dump your table to Comma Separated File and fed that file
to solr?
Ahmet
On Friday, April 4, 2014 5:10 PM, Candygram For Mongo
<candygram.for.mo...@gmail.com> wrote:
The ramBufferSizeMB was set to 6MB only on the test system to make the system
crash sooner. In production that tag is commented out which I believe forces
the default value to be used.
On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <iori...@yahoo.com> wrote:
Hi,
>
>out of curiosity, why did you set ramBufferSizeMB to 6?
>
>Ahmet
>
>
>
>
>
>On Friday, April 4, 2014 3:27 AM, Candygram For Mongo
><candygram.for.mo...@gmail.com> wrote:
>*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
>
>*SOLR/Lucene version: *4.2.1*
>
>
>*JVM version:
>
>Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>
>Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>
>
>
>*Indexer startup command:
>
>set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>
>
>
>java " %JVMARGS% ^
>
>-Dcom.sun.management.jmxremote.port=1092 ^
>
>-Dcom.sun.management.jmxremote.ssl=false ^
>
>-Dcom.sun.management.jmxremote.authenticate=false ^
>
>-jar start.jar
>
>
>
>*SOLR indexing HTTP parameters request:
>
>webapp=/solr path=/dataimport
>params={clean=false&command=full-import&wt=javabin&version=2}
>
>
>
>We are getting a Java heap OOM exception when indexing (updating) 27
>million records. If we increase the Java heap memory settings the problem
>goes away but we believe the problem has not been fixed and that we will
>eventually get the same OOM exception. We have other processes on the
>server that also require resources so we cannot continually increase the
>memory settings to resolve the OOM issue. We are trying to find a way to
>configure the SOLR instance to reduce or preferably eliminate the
>possibility of an OOM exception.
>
>
>
>We can reproduce the problem on a test machine. We set the Java heap
>memory size to 64MB to accelerate the exception. If we increase this
>setting the same problems occurs, just hours later. In the test
>environment, we are using the following parameters:
>
>
>
>JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>
>
>
>Normally we use the default solrconfig.xml file with only the following jar
>file references added:
>
>
>
><lib path="../../../../default/lib/common.jar" />
>
><lib path="../../../../default/lib/webapp.jar" />
>
><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>
>
>
>Using these values and trying to index 6 million records from the database,
>the Java Heap Out of Memory exception is thrown very quickly.
>
>
>
>We were able to complete a successful indexing by further modifying the
>solrconfig.xml and removing all or all but one <copyfield> tags from the
>schema.xml file.
>
>
>
>The following solrconfig.xml values were modified:
>
>
>
><ramBufferSizeMB>6</ramBufferSizeMB>
>
>
>
><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>
><int name="maxMergeAtOnce">2</int>
>
><int name="maxMergeAtOnceExplicit">2</int>
>
><int name="segmentsPerTier">10</int>
>
><int name="maxMergedSegmentMB">150</int>
>
></mergePolicy>
>
>
>
><autoCommit>
>
><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
>
><openSearcher>false</openSearcher>
>
></autoCommit>
>
>
>
>Using our customized schema.xml file with two or more <copyfield> tags, the
>OOM exception is always thrown. Based on the errors, the problem occurs
>when the process was trying to do the merge. The error is provided below:
>
>
>
>Exception in thread "Lucene Merge Thread #156"
>org.apache.lucene.index.MergePolicy$MergeException:
>java.lang.OutOfMemoryError: Java heap space
>
> at
>org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>
> at
>org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>
>Caused by: java.lang.OutOfMemoryError: Java heap space
>
> at
>org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>
> at
>org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>
> at
>org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>
> at
>org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>
> at
>org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>
> at
>org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>
> at
>org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>
> at
>org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>
> at
>org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>
> at
>org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>
>Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>
>SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
>hit an OutOfMemoryError; cannot commit
>
> at
>org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>
> at
>org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>
> at
>org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>
> at
>org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>
> at
>org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>
> at
>org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>
> at
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>
> at
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:722)
>
>
>
>We think but are not 100% sure that the problem is related to the merge.
>
>
>
>Normally our schema.xml contains a lot of field specifications (like the
>ones seen in the file fragment below):
>
>
>
><copyField source="ADDRESS.RECORD_ID_abc" dest="ADDRESS.RECORD_ID.case_abc"
>/>
>
><copyField source="ADDRESS.RECORD_ID_abc"
>dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>
><copyField source="ADDRESS.RECORD_ID_abc"
>dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>
>
>
>In tests using the default file schema.xml and no <copyfield> tags,
>indexing completed successfully. 6 million records produced a 900 MB data
>directory.
>
>
>
>When I included just one <copyfield> tag, indexing completed successfully. 6
>million records produced a 990 MB data directory (90 MB bigger).
>
>
>
>When I included just two <copyfield> tags, the index crashed with an OOM
>exception.
>
>
>
>Changing parameters like maxMergedSegmentMB or maxDocs, only postponed the
>crash.
>
>
>
>The net of our test results I as follows:
>
>
>
>*solrconfig.xml*
>
>*schema.xml*
>
>*result*
>
>
>default plus only jar references
>
>default (no copyfield tags)
>
>success
>
>default plus only jar references
>
>modified with one copyfield tag
>
>success
>
>default plus only jar references
>
>modified with two copyfield tags
>
>crash
>
>additional modified settings
>
>default (no copyfield tags)
>
>success
>
>additional modified settings
>
>modified with one copyfield tag
>
>success
>
>additional modified settings
>
>modified with two copyfield tags
>
>crash
>
>
>
>
>
>Our question is, what can we do to eliminate these OOM exceptions?
>
>
