Hi all,
I'm experimenting with memory use in Solr 4.10.2. Our index is currently
about 250GB and we have allocated 4GB to solr. I'm getting OOM errors:
"java.lang.OutOfMemoryError: Java heap space at
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray."
(More details appended below)
I'm not up to speed yet on the data structures the Solr 4 codecs use. My
first guess is that this is occurring during a large merge where the size
of the array gets huge. (We have about 3 billion unique terms per shard)
Can someone tell me what the FreqProxPostingsArray is and whether my guess
is in the right ballpark?
I can't seem to find anything about these errors in the indexwriter
(InfoStream) log. What should I be looking for? Should I set the
indexwriter log level to something other than INFO?
In Solr 3 we could set the TermIndexInterval to reduce the memory needed by
our huge number of terms. Is there a similar setting for Solr 4?
Tom
-----------------------------------------------
summary error trace:
org.apache.solr.common.SolrException; org.apache.solr.common.SolrException:
Exception writing document id mdp.39015026399660 to the index; possible
analysis error.
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter
is closed
at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:698)
java.lang.OutOfMemoryError: Java heap space at
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.&
