Harish Verma created SOLR-4356:
----------------------------------
Summary: SOLR 4.1 Out Of Memory error After commit of a few
thousand Solr Docs
Key: SOLR-4356
URL: https://issues.apache.org/jira/browse/SOLR-4356
Project: Solr
Issue Type: Improvement
Components: clients - java, Schema and Analysis, Tests
Affects Versions: 4.1
Environment: OS = Ubuntu 12.04
Sun JAVA 7
Max Java Heap Space = 2GB
Apache Tomcat 7
Hardware = {Intel core i3, 2GB RAM}
Average no of fields in a Solr Doc = 100
Reporter: Harish Verma
Fix For: 4.1.1
we are testing solr 4.1 running inside tomcat 7 and java 7 with following
options
JAVA_OPTS="-Xms256m -Xmx2048m -XX:MaxPermSize=1024m -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+ParallelRefProcEnabled
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ubuntu/OOM_HeapDump"
our source code looks like following:
/**** START *****/
int noOfSolrDocumentsInBatch = 0;
for(int i=0 ; i<5000 ; i++) {
SolrInputDocument solrInputDocument = getNextSolrInputDocument();
server.add(solrInputDocument);
noOfSolrDocumentsInBatch += 1;
if(noOfSolrDocumentsInBatch == 10) {
server.commit();
noOfSolrDocumentsInBatch = 0;
}
}
/**** END *****/
the method "getNextSolrInputDocument()" generates a solr document with 100
fields (average). Around 50 of the fields are of "text_general" type.
Some of the "test_general" fields consist of approx 1000 words rest consists of
few words. Ouf of total fields there are around 35-40 multivalued fields (not
of type "text_general").
We are indexing all the fields but storing only 8 fields. Out of these 8 fields
two are string type, five are long and one is boolean. So our index size is
only 394 MB. But the RAM occupied at time of OOM is around 2.5 GB. Why the
memory is so high even though the index size is small?
What is being stored in the memory? Our understanding is that after every
commit documents are flushed to the disk.So nothing should remain in RAM after
commit.
We are using the following settings:
server.commit() set waitForSearcher=true and waitForFlush=true
solrConfig.xml has following properties set:
directoryFactory = solr.MMapDirectoryFactory
maxWarmingSearchers = 1
text_general data type is being used as supplied in the schema.xml with the
solr setup.
maxIndexingThreads = 8(default)
<autoCommit><maxTime>15000</maxTime><openSearcher>false</openSearcher></autoCommit>
We get Java heap Out Of Memory Error after commiting around 3990 solr
documents.Some of the snapshots of memory dump from profiler are attached.
can somebody please suggest what should we do to minimize/optimize the memory
consumption in our case with the reasons?
also suggest what should be optimal values and reason for following parameters
of solrConfig.xml
useColdSearcher - true/false?
maxwarmingsearchers- number
spellcheck-on/off?
omitNorms=true/false?
omitTermFreqAndPositions?
mergefactor? we are using default value 10
java garbage collection tuning parameters ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
