Here is the stack trace:
notice in stack trace * "at
com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)"*
It looks like that its trying to read whole table into memory at a time. n
thts y getting OOM.
Apr 14, 2009 11:15:01 AM org.apache.solr.handler.dataimport.DataImporter
doFullImport
SEVERE: Full Import failed
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:468)
at
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2534)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2159)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2548)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2477)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:741)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:587)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:243)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:207)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:38)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:335)
... 5 more
Apr 14, 2009 11:15:01 AM org.apache.solr.update.DirectUpdateHandler2
rollback
INFO: start rollback
Apr 14, 2009 11:15:01 AM org.apache.solr.update.DirectUpdateHandler2
rollback
INFO: end_rollback
On Tue, Apr 14, 2009 at 11:09 AM, Mani Kumar <manikumarchau...@gmail.com>wrote:
> Hi ILAN:
>
> Only one query is required to generate a document ...
> Here is my data-config.xml
>
> <dataConfig>
> <dataSource type="JdbcDataSource" name="sp"
> driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost/mydb_development"
> user="root" password="******" />
> <document name="items">
> <entity name="item" dataSource="sp" query="select * from items">
> <field column="id" name="id" />
> <field column="title" name="title" />
> </entity>
> </document>
> </dataConfig>
>
> and other useful info:
>
> mysql> select * from items
> +----------+
> | count(*) |
> +----------+
> | 900051 |
> +----------+
> 1 row in set (0.00 sec)
>
> Each record consist of id and title.
>
> id is of type int(11) and title's avg. length is 50 chars.
>
>
> I am using tomcat with solr.
> here is the command i am using to start it
>
> ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>
>
> Thanks! for help. I appreciate it.
>
> -Mani Kumar
>
> On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch <i...@fonz.net> wrote:
>
>> Depending on your dataset and how your queries look you may very likely
>> need to increase to a larger heap size. How many queries and rows are
>> required for each of your documents to be generated?
>>
>> Ilan
>>
>>
>> On 4/13/09 12:21 PM, Mani Kumar wrote:
>>
>>> Hi Shalin:
>>>
>>> Thanks for quick response!
>>>
>>> By defaults it was set to 1.93 MB.
>>> But i also tried it with following command:
>>>
>>> $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>>>
>>> I also tried tricks given on
>>> http://wiki.apache.org/solr/DataImportHandlerFaq page.
>>>
>>> what should i try next ?
>>>
>>> Thanks!
>>> Mani Kumar
>>>
>>> On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar<
>>> shalinman...@gmail.com> wrote:
>>>
>>> On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar<manikumarchau...@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> Hi All,
>>>>> I am trying to setup a Solr instance on my macbook.
>>>>>
>>>>> I get following errors when m trying to do a full db import ... please
>>>>>
>>>> help
>>>>
>>>>> me on this
>>>>>
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
>>>>
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
>>>>
>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>> at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
>>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
>>>>> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
>>>>>
>>>>>
>>>>> How much heap size have you allocated to the jvm?
>>>>
>>>> Also see http://wiki.apache.org/solr/DataImportHandlerFaq
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>>
>>>
>>
>> --
>> Ilan Rabinovitch
>> i...@fonz.net
>>
>> ---
>> SCALE 7x: 2009 Southern California Linux Expo
>> Los Angeles, CA
>> http://www.socallinuxexpo.org
>>
>>
>
