Re: DIH full-import memory issue
Also, which JDBC driver is this? There are quirks with various
drivers, which should be documented on the DataImportHandler page.
On Mon, May 10, 2010 at 9:47 PM, caman wrote:
>
> This may help:
>
> batchSize : The batchsize used in jdbc connection
>
>
>
> http://wiki.apache.org/solr/DataImportHandler#Configuring_DataSources
>
>
>
>
>
>
>
>
>
> From: Geek Gamer [via Lucene]
> [mailto:ml-node+809069-2054572211-124...@n3.nabble.com]
> Sent: Monday, May 10, 2010 9:42 PM
> To: caman
> Subject: DIH full-import memory issue
>
>
>
> Hi,
>
> I am facing issues with DIH fullimport,
>
> I have a database with 3 million records that will translate into index size
>
> of 6GB.
>
> When I am trying to do full import I am getting out of memory error like :
>
> INFO: Starting Full Import
> May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.SolrWriter
> readIndexerProperties
> WARNING: Unable to read: dataimport.properties
> May 10, 2010 11:44:06 PM org.apache.solr.update.DirectUpdateHandler2
> deleteAll
> INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
> May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=1
> commit{dir=/home/search/SOLR/solr/data/index,segFN=segments_1,version=127354
> 9043650,generation=1,filenames=[segments_1]
> May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy
> updateCommits
> INFO: newest commit = 1273549043650
> May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
>
> call
> INFO: Creating a connection for entity offer with URL:
> jdbc:mysql://domU-12-31-39-10-59-01.compute-1.internal/jounce1
> May 10, 2010 11:44:07 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
>
> call
> INFO: Time taken for getConnection(): 301
>
>
>
> Exception in thread "Timer-1" java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.newValueIterator(HashMap.java:843)
> at java.util.HashMap$Values.iterator(HashMap.java:910)
> at
> org.mortbay.jetty.servlet.HashSessionManager.scavenge(HashSessionManager.jav
> a:180)
> at
> org.mortbay.jetty.servlet.HashSessionManager.access$000(HashSessionManager.j
> ava:36)
> at
> org.mortbay.jetty.servlet.HashSessionManager$1.run(HashSessionManager.java:1
> 44)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> May 10, 2010 11:54:54 PM org.apache.solr.handler.dataimport.DataImporter
> doFullImport
> SEVERE: Full Import failed
> org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:
> 424)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242
> )
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.ja
> va:331)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389
> )
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1621)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398)
> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816)
> at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
> at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
> at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
> at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:734)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(J
> dbcDataSource.java:246)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav
> a:210)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav
> a:39)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityPro
> cessor.java:58)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProce
> ssor.java:71)
> at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProc
> essorWrapper.java:237)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:
> 357)
> ... 5 more
> May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2
> rollback
> INFO: start rollback
> May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2
> rollback
> INFO: end_rollback
>
>
>
>
> I tried allocating 4 Gigs of memory to the VM but no luck.
> Are the records cached before indexing or streamed?
> any pointers to documents?
>
> thanks in anticipation,
> umar
>
>
>
> _
>
> View message @
> http://lucene.472066.n3.nabble.com/DIH-full-import-memory-issue-t
RE: DIH full-import memory issue
This may help:
batchSize : The batchsize used in jdbc connection
http://wiki.apache.org/solr/DataImportHandler#Configuring_DataSources
From: Geek Gamer [via Lucene]
[mailto:ml-node+809069-2054572211-124...@n3.nabble.com]
Sent: Monday, May 10, 2010 9:42 PM
To: caman
Subject: DIH full-import memory issue
Hi,
I am facing issues with DIH fullimport,
I have a database with 3 million records that will translate into index size
of 6GB.
When I am trying to do full import I am getting out of memory error like :
INFO: Starting Full Import
May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
WARNING: Unable to read: dataimport.properties
May 10, 2010 11:44:06 PM org.apache.solr.update.DirectUpdateHandler2
deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=/home/search/SOLR/solr/data/index,segFN=segments_1,version=127354
9043650,generation=1,filenames=[segments_1]
May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1273549043650
May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity offer with URL:
jdbc:mysql://domU-12-31-39-10-59-01.compute-1.internal/jounce1
May 10, 2010 11:44:07 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 301
Exception in thread "Timer-1" java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.newValueIterator(HashMap.java:843)
at java.util.HashMap$Values.iterator(HashMap.java:910)
at
org.mortbay.jetty.servlet.HashSessionManager.scavenge(HashSessionManager.jav
a:180)
at
org.mortbay.jetty.servlet.HashSessionManager.access$000(HashSessionManager.j
ava:36)
at
org.mortbay.jetty.servlet.HashSessionManager$1.run(HashSessionManager.java:1
44)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
May 10, 2010 11:54:54 PM org.apache.solr.handler.dataimport.DataImporter
doFullImport
SEVERE: Full Import failed
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:
424)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242
)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.ja
va:331)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389
)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1621)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:734)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(J
dbcDataSource.java:246)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav
a:210)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav
a:39)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityPro
cessor.java:58)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProce
ssor.java:71)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProc
essorWrapper.java:237)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:
357)
... 5 more
May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2
rollback
INFO: start rollback
May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2
rollback
INFO: end_rollback
I tried allocating 4 Gigs of memory to the VM but no luck.
Are the records cached before indexing or streamed?
any pointers to documents?
thanks in anticipation,
umar
_
View message @
http://lucene.472066.n3.nabble.com/DIH-full-import-memory-issue-tp809069p809
069.html
To start a new topic under Solr - User, email
ml-node+472068-464289649-124...@n3.nabble.com
To unsubscribe from Solr - User, click
< (link removed)
GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx> here.
--
View this message in context:
http://lucene.472066.n3.nabble.com/DIH-full-import-memory-issue-tp8090
