Re: Full Indexing is Causing a Java Heap Out of Memory Exception

I wanted to take a moment and say thank you for your help. We haven't solved the problem yet but it seems like we may be on the path. Responses to your questions below: 1) We are using settings of 6GBs for -Xmx and -Xms on a production server where this process is failing on about 30 million

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

Hi, I had similar problems before. We were trying to do same thing as you, fetching too many small documents from Oracle with dih. We were getting  Caused by: java.sql.SQLException: ORA-01652: unable to extend temp segment by 128 in tablespace TS_TEMP ORA-06512: at IZCI.GET_FEED_KEYWORDS, line

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

Hi, Now we have a more informative error :  org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space         at

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

The ramBufferSizeMB was set to 6MB only on the test system to make the system crash sooner. In production that tag is commented out which I believe forces the default value to be used. On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan iori...@yahoo.com wrote: Hi, out of curiosity, why did you

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

Hi, Which database are you using? Can you send us data-config.xml?  What happens when you use default merge policy settings? What happens when you dump your table to Comma Separated File and fed that file to solr? Ahmet On Friday, April 4, 2014 5:10 PM, Candygram For Mongo

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

In this case we are indexing an Oracle database. We do not include the data-config.xml in our distribution. We store the database information in the database.xml file. I have attached the database.xml file. When we use the default merge policy settings, we get the same results. We have not

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

In case the attached database.xml file didn't show up, I have pasted in the contents below: dataConfig dataSource name=org_only type=JdbcDataSource driver=oracle.jdbc.OracleDriver url=jdbc:oracle:thin:@test2.abc.com:1521:ORCL user=admin password=admin readOnly=false batchSize=100 / document

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

Hi, Can you remove auto commit for bulk import. Commit at the very end? Ahmet On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo candygram.for.mo...@gmail.com wrote: In case the attached database.xml file didn't show up, I have pasted in the contents below: dataConfig dataSource

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

We would be happy to try that. That sounds counter intuitive for the high volume of records we have. Can you help me understand how that might solve our problem? On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan iori...@yahoo.com wrote: Hi, Can you remove auto commit for bulk import. Commit at

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

I might have forgot to mention that we are using the DataImportHandler. I think we know how to remove auto commit. How would we force a commit at the end? On Fri, Apr 4, 2014 at 3:18 PM, Candygram For Mongo candygram.for.mo...@gmail.com wrote: We would be happy to try that. That sounds

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

Hi, This may not solve your problem but generally it is recommended to disable auto commit and transaction logs for bulk indexing. And issue one commit at the very end. Do you tlogs enabled? I see commit failed in the error message thats why I am offering this. And regarding comma separated

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

Hi, To disable auto commit remove both autoCommit and autoSoftCommit parts/definitions from solrconfig.xml To disable tlog remove      updateLog       str name=dir${solr.ulog.dir:}/str     /updateLog from solrconfig.xml To commit at the end use commit=true parameter.

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

Guessing that the attachments won't work, I am pasting one file in each of four separate emails. database.xml dataConfig dataSource name=org_only type=JdbcDataSource driver=oracle.jdbc.OracleDriver url=jdbc:oracle:thin:@test.abcdata.com:1521:ORCL user=admin password=admin readOnly=false /

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

error.txt below Java Platform Detected x64 Java Platform Detected -XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m -XX:+HeapDumpOnOutOfMemoryError -XX:+CreateMinidumpOnCrash 2014-04-04 15:49:43.341:INFO:oejs.Server:jetty-8.1.8.v20121106 2014-04-04

Full Indexing is Causing a Java Heap Out of Memory Exception

*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception *SOLR/Lucene version: *4.2.1* *JVM version: Java(TM) SE Runtime Environment (build 1.7.0_07-b11) Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode) *Indexer startup command: set

Re: Full Indexing is Causing a Java Heap Out of Memory Exception

Hi, out of curiosity, why did you set ramBufferSizeMB to 6?  Ahmet On Friday, April 4, 2014 3:27 AM, Candygram For Mongo candygram.for.mo...@gmail.com wrote: *Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception *SOLR/Lucene version: *4.2.1* *JVM version: Java(TM) SE