We have sufficient RAM on machine ..64 GB and we have given JVM 32 GB of
memory. The machine runs Indexing primarily.
The JVM doesn't run out of memory. It is the particular IndexWriterSolrCore
which has .. May be we have specified too low a memory for IndexWriter ..
We index mainly product data and use DIH to pull data from downstream
services. Autocommiit is off. The commit is infrequent for legacy
reasons.. 1 commit in 2-3 hrs. It it makes a difference, then, a Core can
have more than10 lakh documents uncommitted at a time. IndexWriter has a
memory of 100 MB
We ran with same config on Solr 3.5 and we never ran out of Memory.
But then, I hadn't tried hard commits on Solr 3.5.
Data-Source Entry :
<dataConfig>
<dataSource name="products" type="MultiSPCMSProductsDataSource"
spCmsHost="$config.spCmsHost" spCmsPort="$config.spCmsPort"
spCmsTimeout="30000" cmsBatchSize="200" psURL="$config.psUrl"
autoCommit="false"/>
<document name="products">
<entity name="item" pk="id"
transformer="w3.solr.transformers.GenericProductsTransformer"
dataSource="products">
</entity>
</document>
</dataConfig>
IndexConfig.
<ramBufferSizeMB>100</ramBufferSizeMB>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>50000</maxFieldLength>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
On Tue, May 21, 2013 at 7:07 PM, Jack Krupansky <j...@basetechnology.com>wrote:
> Try again on a machine with more memory. Or did you do that already?
>
> -- Jack Krupansky
>
> -----Original Message----- From: Umesh Prasad
> Sent: Tuesday, May 21, 2013 1:57 AM
> To: solr-user@lucene.apache.org
> Subject: Hard Commit giving OOM Error on Index Writer in Solr 4.2.1
>
>
> Hi All,
> I am hitting an OOM error while trying to do an hard commit on one of
> the cores.
>
> Transaction log dir is Empty and DIH shows indexing going on for 13 hrs..
>
> *Indexing since 13h 22m 22s*
> Requests: 5,211,392 (108/s), Fetched: 1,902,792 (40/s), Skipped: 106,853,
> Processed: 1,016,696 (21/s)
> Started: about 13 hours ago
>
>
>
> response>
> <lst name="responseHeader"><int name="status">500</int><int
> name="QTime">4</int></lst><lst name="error"><str name="msg">this writer hit
> an OutOfMemoryError; cannot commit</str><str
> name="trace">java.lang.**IllegalStateException: this writer hit an
> OutOfMemoryError; cannot commit
> at
> org.apache.lucene.index.**IndexWriter.**prepareCommitInternal(**
> IndexWriter.java:2661)
> at
> org.apache.lucene.index.**IndexWriter.commitInternal(**
> IndexWriter.java:2827)
> at org.apache.lucene.index.**IndexWriter.commit(**
> IndexWriter.java:2807)
> at
> org.apache.solr.update.**DirectUpdateHandler2.commit(**
> DirectUpdateHandler2.java:536)
> at
> org.apache.solr.update.**processor.RunUpdateProcessor.**processCommit(**
> RunUpdateProcessorFactory.**java:95)
> at
> org.apache.solr.update.**processor.**UpdateRequestProcessor.**
> processCommit(**UpdateRequestProcessor.java:**64)
> at
> org.apache.solr.update.**processor.**DistributedUpdateProcessor.**
> processCommit(**DistributedUpdateProcessor.**java:1055)
> at
> org.apache.solr.update.**processor.LogUpdateProcessor.**processCommit(**
> LogUpdateProcessorFactory.**java:157)
> at
> org.apache.solr.handler.**RequestHandlerUtils.**handleCommit(**
> RequestHandlerUtils.java:69)
> at
> org.apache.solr.handler.**ContentStreamHandlerBase.**handleRequestBody(**
> ContentStreamHandlerBase.java:**68)
> at
> org.apache.solr.handler.**RequestHandlerBase.**handleRequest(**
> RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.**execute(SolrCore.java:1817)
> at
> org.apache.solr.servlet.**SolrDispatchFilter.execute(**
> SolrDispatchFilter.java:639)
> at
> org.apache.solr.servlet.**SolrDispatchFilter.doFilter(**
> SolrDispatchFilter.java:345)
> at
> org.apache.solr.servlet.**SolrDispatchFilter.doFilter(**
> SolrDispatchFilter.java:141)
> at
> org.apache.catalina.core.**ApplicationFilterChain.**internalDoFilter(**
> ApplicationFilterChain.java:**235)
> at
> org.apache.catalina.core.**ApplicationFilterChain.**doFilter(**
> ApplicationFilterChain.java:**206)
> at
> org.apache.catalina.core.**StandardWrapperValve.invoke(**
> StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.**StandardContextValve.invoke(**
> StandardContextValve.java:191)
> at
> org.apache.catalina.core.**StandardHostValve.invoke(**
> StandardHostValve.java:127)
> at
> org.apache.catalina.valves.**ErrorReportValve.invoke(**
> ErrorReportValve.java:102)
> at
> org.apache.catalina.core.**StandardEngineValve.invoke(**
> StandardEngineValve.java:109)
> at
> org.apache.catalina.valves.**AccessLogValve.invoke(**
> AccessLogValve.java:554)
> at
> org.apache.catalina.connector.**CoyoteAdapter.service(**
> CoyoteAdapter.java:298)
> at
> org.apache.coyote.http11.**Http11Processor.process(**
> Http11Processor.java:859)
> at
> org.apache.coyote.http11.**Http11Protocol$**Http11ConnectionHandler.**
> process(Http11Protocol.java:**588)
> at
> org.apache.tomcat.util.net.**JIoEndpoint$Worker.run(**
> JIoEndpoint.java:489)
> at java.lang.Thread.run(Thread.**java:662)
>
>
>
>
> --
> ---
> Thanks & Regards
> Umesh Prasad
>
--
---
Thanks & Regards
Umesh Prasad
