I'm trying to index 5M Documents and currently indexing takes 13 hours. My data source is SQL db and I've verified out of 13 hours only 30 mins is spent in fetching data so SQL is not the bottleneck. The complete index size is 40GB.
My application runs on Tomcat with 2GB JVM space. Just after the indexing process is complete, when I try to run a simple query, the application hits OutOfMemoryError: Java Heap Space The InfoReader log reports 'hit exception during NRT Reader' <http://lucene.472066.n3.nabble.com/file/n4345589/exception_during_nrt_reader.png> Also around the same time frame the InfoReader reports following for updateMergeThreads <http://lucene.472066.n3.nabble.com/file/n4345589/updateMergeThreads.png> I analyzed the logs and found that the merging process is painfully slow. I'm using the Default IndexWriterConfig settings. I've tried indexing same 5m Docs on 3GB JVM with a SerialMergeScheduler with setRAMBufferSizeMB=48 and setMaxMergedSizeMB=2G in TieredMergePolicy which takes 14Hours. Increasing the tomcat jvm space may not always be an option, so is there any way to tune the Merging process to make it faster and also not run out of memory. -- View this message in context: http://lucene.472066.n3.nabble.com/lucene-6-3-0-hit-tragic-OutOfMemoryError-inside-getReader-tp4345589.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
