On 5/21/2013 5:14 PM, Umesh Prasad wrote:
We have sufficient RAM on machine ..64 GB and we have given JVM 32 GB of
memory. The machine runs Indexing primarily.
The JVM doesn't run out of memory. It is the particular IndexWriterSolrCore
which has .. May be we have specified too low a memory for IndexWriter ..
We index mainly product data and use DIH to pull data from downstream
services. Autocommiit is off. The commit is infrequent for legacy
reasons.. 1 commit in 2-3 hrs. It it makes a difference, then, a Core can
have more than10 lakh documents uncommitted at a time. IndexWriter has a
memory of 100 MB
We ran with same config on Solr 3.5 and we never ran out of Memory.
But then, I hadn't tried hard commits on Solr 3.5.
Hard commits are the only kind of commits that Solr 3.x has. It's soft
commits that are new with 4.x.
Data-Source Entry :
<dataConfig>
<dataSource name="products" type="MultiSPCMSProductsDataSource"
This appears to be using a custom data source, not one of the well-known
types. If it had been JDBC, I would be saying that your JDBC driver is
trying to cache the entire result set in RAM. With a MySQL data source,
a batchSize of -1 fixes this problem, by internally changing the JDBC
fetchSize to Integer.MIN_VALUE. Other databases have different mechanisms.
With this data source, I have no idea at all how to make sure that it
doesn't cache all results in RAM. It might be that the combination of
the new Solr and this custom data source causes a memory leak, something
that doesn't happen with the old Solr version.
You said that the transaction log directory is empty. That rules out
one possibility, which would be solved by the autoCommit settings on
this page:
http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_startup
Aside from the memory leak idea, or possibly having your entire source
data cached in RAM, I have no idea what's happening here.
Thanks,
Shawn
