We would be happy to try that. That sounds counter intuitive for the high
volume of records we have. Can you help me understand how that might solve
our problem?
On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <iori...@yahoo.com> wrote:
> Hi,
>
> Can you remove auto commit for bulk import. Commit at the very end?
>
> Ahmet
>
>
>
> On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
> candygram.for.mo...@gmail.com> wrote:
> In case the attached database.xml file didn't show up, I have pasted in the
> contents below:
>
> <dataConfig>
> <dataSource
> name="org_only"
> type="JdbcDataSource"
> driver="oracle.jdbc.OracleDriver"
> url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
> user="admin"
> password="admin"
> readOnly="false"
> batchSize="100"
> />
> <document>
>
>
> <entity name="full-index" query="
> select
>
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
> as SOLR_ID,
>
> 'ORCL.ADDRESS_ACCT_ALL'
> as SOLR_CATEGORY,
>
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
> ADDRESSALLROWID,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
> ADDRESSALLADDRTYPECD,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
> ADDRESSALLLONGITUDE,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
> ADDRESSALLLATITUDE,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
> ADDRESSALLADDRNAME,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
> ADDRESSALLCITY,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
> ADDRESSALLSTATE,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
> ADDRESSALLEMAILADDR
>
> from ORCL.ADDRESS_ACCT_ALL
> " >
>
> <field column="SOLR_ID" name="id" />
> <field column="SOLR_CATEGORY" name="category" />
> <field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
> <field column="ADDRESSALLADDRTYPECD"
> name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
> <field column="ADDRESSALLLONGITUDE" name="ADDRESS_ACCT_ALL.LONGITUDE_abc"
> />
> <field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
> <field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc" />
> <field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
> <field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
> <field column="ADDRESSALLEMAILADDR" name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
> />
>
> </entity>
>
>
>
> <!-- Varaibles -->
> <!-- '${dataimporter.last_index_time}' -->
> </document>
> </dataConfig>
>
>
>
>
>
>
> On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
> candygram.for.mo...@gmail.com> wrote:
>
> > In this case we are indexing an Oracle database.
> >
> > We do not include the data-config.xml in our distribution. We store the
> > database information in the database.xml file. I have attached the
> > database.xml file.
> >
> > When we use the default merge policy settings, we get the same results.
> >
> >
> >
> > We have not tried to dump the table to a comma separated file. We think
> > that dumping this size table to disk will introduce other memory problems
> > with big file management. We have not tested that case.
> >
> >
> > On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <iori...@yahoo.com> wrote:
> >
> >> Hi,
> >>
> >> Which database are you using? Can you send us data-config.xml?
> >>
> >> What happens when you use default merge policy settings?
> >>
> >> What happens when you dump your table to Comma Separated File and fed
> >> that file to solr?
> >>
> >> Ahmet
> >>
> >> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
> >> candygram.for.mo...@gmail.com> wrote:
> >>
> >> The ramBufferSizeMB was set to 6MB only on the test system to make the
> >> system crash sooner. In production that tag is commented out which
> >> I believe forces the default value to be used.
> >>
> >>
> >>
> >>
> >> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <iori...@yahoo.com> wrote:
> >>
> >> Hi,
> >> >
> >> >out of curiosity, why did you set ramBufferSizeMB to 6?
> >> >
> >> >Ahmet
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
> >> candygram.for.mo...@gmail.com> wrote:
> >> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
> Exception
> >> >
> >> >*SOLR/Lucene version: *4.2.1*
> >> >
> >> >
> >> >*JVM version:
> >> >
> >> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
> >> >
> >> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
> >> >
> >> >
> >> >
> >> >*Indexer startup command:
> >> >
> >> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
> >> >
> >> >
> >> >
> >> >java " %JVMARGS% ^
> >> >
> >> >-Dcom.sun.management.jmxremote.port=1092 ^
> >> >
> >> >-Dcom.sun.management.jmxremote.ssl=false ^
> >> >
> >> >-Dcom.sun.management.jmxremote.authenticate=false ^
> >> >
> >> >-jar start.jar
> >> >
> >> >
> >> >
> >> >*SOLR indexing HTTP parameters request:
> >> >
> >> >webapp=/solr path=/dataimport
> >> >params={clean=false&command=full-import&wt=javabin&version=2}
> >> >
> >> >
> >> >
> >> >We are getting a Java heap OOM exception when indexing (updating) 27
> >> >million records. If we increase the Java heap memory settings the
> >> problem
> >> >goes away but we believe the problem has not been fixed and that we
> will
> >> >eventually get the same OOM exception. We have other processes on the
> >> >server that also require resources so we cannot continually increase
> the
> >> >memory settings to resolve the OOM issue. We are trying to find a way
> to
> >> >configure the SOLR instance to reduce or preferably eliminate the
> >> >possibility of an OOM exception.
> >> >
> >> >
> >> >
> >> >We can reproduce the problem on a test machine. We set the Java heap
> >> >memory size to 64MB to accelerate the exception. If we increase this
> >> >setting the same problems occurs, just hours later. In the test
> >> >environment, we are using the following parameters:
> >> >
> >> >
> >> >
> >> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
> >> >
> >> >
> >> >
> >> >Normally we use the default solrconfig.xml file with only the following
> >> jar
> >> >file references added:
> >> >
> >> >
> >> >
> >> ><lib path="../../../../default/lib/common.jar" />
> >> >
> >> ><lib path="../../../../default/lib/webapp.jar" />
> >> >
> >> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
> >> >
> >> >
> >> >
> >> >Using these values and trying to index 6 million records from the
> >> database,
> >> >the Java Heap Out of Memory exception is thrown very quickly.
> >> >
> >> >
> >> >
> >> >We were able to complete a successful indexing by further modifying the
> >> >solrconfig.xml and removing all or all but one <copyfield> tags from
> the
> >> >schema.xml file.
> >> >
> >> >
> >> >
> >> >The following solrconfig.xml values were modified:
> >> >
> >> >
> >> >
> >> ><ramBufferSizeMB>6</ramBufferSizeMB>
> >> >
> >> >
> >> >
> >> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> >> >
> >> ><int name="maxMergeAtOnce">2</int>
> >> >
> >> ><int name="maxMergeAtOnceExplicit">2</int>
> >> >
> >> ><int name="segmentsPerTier">10</int>
> >> >
> >> ><int name="maxMergedSegmentMB">150</int>
> >> >
> >> ></mergePolicy>
> >> >
> >> >
> >> >
> >> ><autoCommit>
> >> >
> >> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this
> -- >
> >> >
> >> ><openSearcher>false</openSearcher>
> >> >
> >> ></autoCommit>
> >> >
> >> >
> >> >
> >> >Using our customized schema.xml file with two or more <copyfield> tags,
> >> the
> >> >OOM exception is always thrown. Based on the errors, the problem
> occurs
> >> >when the process was trying to do the merge. The error is provided
> >> below:
> >> >
> >> >
> >> >
> >> >Exception in thread "Lucene Merge Thread #156"
> >> >org.apache.lucene.index.MergePolicy$MergeException:
> >> >java.lang.OutOfMemoryError: Java heap space
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
> >> >
> >> >Caused by: java.lang.OutOfMemoryError: Java heap space
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
> >> >
> >> > at
> >>
> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
> >> >
> >> > at
> >> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
> >> >
> >> > at
> >> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
> >> >
> >> > at
> >> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
> >> >
> >> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
> >> >
> >> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
> writer
> >> >hit an OutOfMemoryError; cannot commit
> >> >
> >> > at
> >> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
> >> >
> >> > at
> >>
> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
> >> >
> >> > at
> >> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
> >> >
> >> > at
> >>
> >>
> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
> >> >
> >> > at
> >> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> >> >
> >> > at
> >> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> >
> >> > at
> >> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >> >
> >> > at
> >> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >> >
> >> > at
> >>
> >>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >> >
> >> > at
> >>
> >>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >> >
> >> > at
> >>
> >>
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >> >
> >> > at
> >>
> >>
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >> >
> >> > at java.lang.Thread.run(Thread.java:722)
> >> >
> >> >
> >> >
> >> >We think but are not 100% sure that the problem is related to the
> merge.
> >> >
> >> >
> >> >
> >> >Normally our schema.xml contains a lot of field specifications (like
> the
> >> >ones seen in the file fragment below):
> >> >
> >> >
> >> >
> >> ><copyField source="ADDRESS.RECORD_ID_abc"
> >> dest="ADDRESS.RECORD_ID.case_abc"
> >> >/>
> >> >
> >> ><copyField source="ADDRESS.RECORD_ID_abc"
> >> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
> >> >
> >> ><copyField source="ADDRESS.RECORD_ID_abc"
> >> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
> >> >
> >> >
> >> >
> >> >In tests using the default file schema.xml and no <copyfield> tags,
> >> >indexing completed successfully. 6 million records produced a 900 MB
> >> data
> >> >directory.
> >> >
> >> >
> >> >
> >> >When I included just one <copyfield> tag, indexing completed
> >> successfully. 6
> >> >million records produced a 990 MB data directory (90 MB bigger).
> >> >
> >> >
> >> >
> >> >When I included just two <copyfield> tags, the index crashed with an
> OOM
> >> >exception.
> >> >
> >> >
> >> >
> >> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed
> >> the
> >> >crash.
> >> >
> >> >
> >> >
> >> >The net of our test results I as follows:
> >> >
> >> >
> >> >
> >> >*solrconfig.xml*
> >> >
> >> >*schema.xml*
> >> >
> >> >*result*
> >> >
> >> >
> >> >default plus only jar references
> >> >
> >> >default (no copyfield tags)
> >> >
> >> >success
> >> >
> >> >default plus only jar references
> >> >
> >> >modified with one copyfield tag
> >> >
> >> >success
> >> >
> >> >default plus only jar references
> >> >
> >> >modified with two copyfield tags
> >> >
> >> >crash
> >> >
> >> >additional modified settings
> >> >
> >> >default (no copyfield tags)
> >> >
> >> >success
> >> >
> >> >additional modified settings
> >> >
> >> >modified with one copyfield tag
> >> >
> >> >success
> >> >
> >> >additional modified settings
> >> >
> >> >modified with two copyfield tags
> >> >
> >> >crash
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >Our question is, what can we do to eliminate these OOM exceptions?
> >> >
> >> >
> >>
> >
> >
>
>
