DIH still running out of memory for me, with Full Import on a database of
size 1.5 GB.
Solr version: 3_5_0
Note that I have already added batchSize="-1" but getting same error.
Sharing my DIH config below.
============================================
<dataConfig>
<dataSource type="JdbcDataSource"
name="jdbc"
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/ib"
user="root"
password="root"
batchSize="-1"
/>
<document name="content">
<entity
name="issue"
dataSource="jdbc"
transformer="RegexTransformer,DateFormatTransformer,
TemplateTransformer"
pk="id"
query="
select
ib_issue.`_id` as id,
ib_issue.`_issue_title` as issueTitle,
ib_issue.`_issue_descr` as issueDescr,
createdBy.`_name` as issueCreatedByName,
createdBy.`_email` as issueCreatedByEmail
from
`ib_issue`
inner join `ib_user` as createdBy
on createdBy.`_id` = ib_issue.`_created_by_user_id`
group by ib_issue.`_id` "
>
</entity>
</document>
</dataConfig>
Please find the error trace below
===================================
2012-02-20 19:04:40.531:INFO::Started SocketConnector@0.0.0.0:8983
Feb 20, 2012 7:04:57 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select params={command=status&qt=/dih_ib_jdbc}
status=0 QTime=0
Feb 20, 2012 7:04:58 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select
params={command=show-config&qt=/dih_ib_jdbc} status=0 QTime=0
Feb 20, 2012 7:05:30 PM org.apache.solr.handler.dataimport.DataImporter
doFullImport
INFO: Starting Full Import
Feb 20, 2012 7:05:30 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/dih_ib_jdbc params={command=full-import}
status=0 QTime=0
Feb 20, 2012 7:05:30 PM org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
INFO: Read dih_ib_jdbc.properties
Feb 20, 2012 7:05:30 PM org.apache.solr.update.DirectUpdateHandler2
deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
Feb 20, 2012 7:05:30 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=E:\workspace\solr_3_5_0\example\solr\data\index,segFN=segments_1,version=1329744880204,generation=1,filenames=[segments_1]
Feb 20, 2012 7:05:30 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1329744880204
Feb 20, 2012 7:05:30 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity issue with URL:
jdbc:mysql://localhost:3306/issueburner
Feb 20, 2012 7:05:30 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 172
Feb 20, 2012 7:07:45 PM org.apache.solr.common.SolrException log
SEVERE: Full Import
failed:org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:669)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:268)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:187)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:359)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:427)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.util.UnicodeUtil.UTF16toUTF8(UnicodeUtil.java:377)
at
org.apache.lucene.store.DataOutput.writeString(DataOutput.java:103)
at
org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:200)
at
org.apache.lucene.index.StoredFieldsWriterPerThread.addField(StoredFieldsWriterPerThread.java:58)
at
org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:265)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:766)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2327)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2299)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:115)
at
org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:73)
at
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:293)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:636)
... 5 more
Feb 20, 2012 7:07:45 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback
Feb 20, 2012 7:07:45 PM org.apache.solr.update.DirectUpdateHandler2 rollback
--
View this message in context:
http://lucene.472066.n3.nabble.com/DataImportHandler-running-out-of-memory-tp490797p3760755.html
Sent from the Solr - User mailing list archive at Nabble.com.
