Re: Out of memory error during full import

[Shawn Heisey]

On 2/4/2016 12:18 AM, Srinivas Kashyap wrote:
> I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the 
> child entities in data-config.xml. When i try to do full import, i'm getting 
> OutOfMemory error(Java Heap Space). I increased the HEAP allocation to the 
> maximum extent possible. Is there a workaround to do initial data load 
> without running into this error?
>
> I found that 'batchSize=-1' parameter needs to be specified in the datasource 
> for MySql, is there a way to specify for others Databases as well?

Setting batchSize to -1 in the DIH config translates to a 'setFetchSize'
on the JDBC object of Integer.MIN_VALUE.  This is how to turn on result
streaming in MySQL.

The method for doing this with other JDBC implementations is likely to
be different.  The Microsoft driver for SQL Server uses a URL parameter,
and newer versions of that particular driver have the streaming behavior
as default.  I have no idea how to do it for any other driver, you would
need to ask the author of the driver.

When you turn on caching (SortedMapBackedCache), you are asking Solr to
put all of the data received into memory -- very similar to what happens
if result streaming is not turned on.  When the SQL result is very
large, this can require a LOT of memory.  In situations like that,
you'll just have to remove the caching.  One alternative to child
entities is to do a query using JOIN in a single entity, so that all the
data you need is returned by a single SQL query, where the heavy lifting
is done by the database server instead of Solr.

The MySQL database that serves as the information source for *my* Solr
index is hundreds of gigabytes in size, so caching it is not possible
for me.  The batchSize=-1 option is the only way to get the import to work.

Thanks,
Shawn

Re: Out of memory error

[Fuad Efendi]

Related: SOLR-846

Sent on the TELUS Mobility network with BlackBerry

-Original Message-
From: Erick Erickson 
Date: Tue, 7 Dec 2010 08:11:41 
To: 
Reply-To: solr-user@lucene.apache.org
Subject: Re: Out of memory error

Have you seen this page? http://wiki.apache.org/solr/DataImportHandlerFaq
<http://wiki.apache.org/solr/DataImportHandlerFaq>See especially batchsize,
but it looks like you're already on to that.

Do you have any idea how big the records are in the database? You might
try adjusting the rambuffersize down, what is it at now?

In general, what are our Solr commit options?

Does anything get to Solr or is the OOM when the SQL is executed?
The first question to answer is whether you index anything at all...

There's a little-know DIH debug page you can access at:
.../solr/admin/dataimport.jsp that might help, and progress can be monitored
at:
.../solr/dataimport

DIH can be "interesting", you get finer control with SolrJ and a direct
JDBC connection. If you don't get anywhere with DIH.

Scattergun response, but things to try...

Best
Erick

On Tue, Dec 7, 2010 at 12:03 AM, sivaprasad wrote:

>
> Hi,
>
> When i am trying to import the data using DIH, iam getting Out of memory
> error.The below are the configurations which i have.
>
> Database:Mysql
> Os:windows
> No Of documents:15525532
> In Db-config.xml i made batch size as "-1"
>
> The solr server is running on Linux machine with tomcat.
> i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M
>
> Can anybody has idea, where the things are going wrong?
>
> Regards,
> JS
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Out of memory error

[Erick Erickson]

Have you seen this page? http://wiki.apache.org/solr/DataImportHandlerFaq
See especially batchsize,
but it looks like you're already on to that.

Do you have any idea how big the records are in the database? You might
try adjusting the rambuffersize down, what is it at now?

In general, what are our Solr commit options?

Does anything get to Solr or is the OOM when the SQL is executed?
The first question to answer is whether you index anything at all...

There's a little-know DIH debug page you can access at:
.../solr/admin/dataimport.jsp that might help, and progress can be monitored
at:
.../solr/dataimport

DIH can be "interesting", you get finer control with SolrJ and a direct
JDBC connection. If you don't get anywhere with DIH.

Scattergun response, but things to try...

Best
Erick

On Tue, Dec 7, 2010 at 12:03 AM, sivaprasad wrote:

>
> Hi,
>
> When i am trying to import the data using DIH, iam getting Out of memory
> error.The below are the configurations which i have.
>
> Database:Mysql
> Os:windows
> No Of documents:15525532
> In Db-config.xml i made batch size as "-1"
>
> The solr server is running on Linux machine with tomcat.
> i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M
>
> Can anybody has idea, where the things are going wrong?
>
> Regards,
> JS
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Out of memory error

[Fuad Efendi]

Batch size "-1"??? Strange but could be a problem. 

Note also you can't provide parameters to default startup.sh command; you 
should modify setenv.sh instead

--Original Message--
From: sivaprasad
To: solr-user@lucene.apache.org
ReplyTo: solr-user@lucene.apache.org
Subject: Out of memory error
Sent: Dec 7, 2010 12:03 AM


Hi,

When i am trying to import the data using DIH, iam getting Out of memory
error.The below are the configurations which i have.

Database:Mysql
Os:windows
No Of documents:15525532
In Db-config.xml i made batch size as "-1"

The solr server is running on Linux machine with tomcat.
i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M

Can anybody has idea, where the things are going wrong?

Regards,
JS


-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
Sent from the Solr - User mailing list archive at Nabble.com.


Sent on the TELUS Mobility network with BlackBerry