Erik, I have seen many posts regarding out of memory error but I am not sure whether they are using cachesqlEntityProcessor..
I want to know if there is a way to flush out the buffer of cache instead of storing everything in cache. I can clearly see the heapsize growing like anything if I use the cachesqlentity processor, trying to figure out if there is a way to resolve this by using any other way other than using this processor. Thanks, Barani Erick Erickson wrote: > > Have you searched the users' list? This question has come up multiple > times > and you'll find your question has probably already been answered. Let us > know if you come up blank... > > Best > Erick > > On Sat, Mar 13, 2010 at 3:56 PM, JavaGuy84 <bbar...@gmail.com> wrote: > >> >> Sorry forgot to attach the error log, >> >> Error Log: >> --------- >> org.apache.solr.handler.dataimport.DataImportHandlerException: >> java.lang.OutOfMe >> moryError: Java heap space >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde >> r.java:650) >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde >> r.java:605) >> at >> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j >> ava:261) >> at >> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java >> :185) >> at >> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo >> rter.java:333) >> at >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j >> ava:391) >> at >> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja >> va:372) >> Caused by: java.lang.OutOfMemoryError: Java heap space >> at java.util.HashMap.<init>(Unknown Source) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g >> etARow(JdbcDataSource.java:281) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a >> ccess$800(JdbcDataSource.java:228) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1 >> .next(JdbcDataSource.java:266) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1 >> .next(JdbcDataSource.java:269) >> at >> org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon >> CachedRows(CachedSqlEntityProcessor.java:70) >> at >> org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData >> (EntityProcessorBase.java:194) >> at >> org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.nextRow(C >> achedSqlEntityProcessor.java:58) >> at >> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent >> ityProcessorWrapper.java:233) >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde >> r.java:579) >> ... 6 more >> Mar 13, 2010 3:52:09 PM org.apache.solr.handler.dataimport.DataImporter >> doFullIm >> port >> SEVERE: Full Import failed >> org.apache.solr.handler.dataimport.DataImportHandlerException: >> java.lang.OutOfMe >> moryError: Java heap space >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde >> r.java:650) >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde >> r.java:605) >> at >> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j >> ava:261) >> at >> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java >> :185) >> at >> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo >> rter.java:333) >> at >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j >> ava:391) >> at >> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja >> va:372) >> Caused by: java.lang.OutOfMemoryError: Java heap space >> at java.util.HashMap.<init>(Unknown Source) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g >> etARow(JdbcDataSource.java:281) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a >> ccess$800(JdbcDataSource.java:228) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1 >> .next(JdbcDataSource.java:266) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1 >> .next(JdbcDataSource.java:269) >> at >> org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon >> CachedRows(CachedSqlEntityProcessor.java:70) >> at >> org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData >> (EntityProcessorBase.java:194) >> at >> org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.nextRow(C >> achedSqlEntityProcessor.java:58) >> at >> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent >> ityProcessorWrapper.java:233) >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde >> r.java:579) >> ... 6 more >> Mar 13, 2010 3:52:09 PM org.apache.solr.update.DirectUpdateHandler2 >> rollback >> INFO: start rollback >> Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space >> at java.util.ArrayList.<init>(Unknown Source) >> at java.util.ArrayList.<init>(Unknown Source) >> at org.mortbay.util.Scanner.reportDifferences(Scanner.java:289) >> at org.mortbay.util.Scanner.scan(Scanner.java:254) >> at org.mortbay.util.Scanner$1.run(Scanner.java:225) >> at java.util.TimerThread.mainLoop(Unknown Source) >> Exception in thread "SocketServerInstance-crprdnii1soe-31000" at >> java.util.Tim >> erThread.run(Unknown Source) >> java.lang.OutOfMemoryError: Java heap space >> at java.nio.CharBuffer.wrap(Unknown Source) >> at sun.nio.cs.StreamEncoder.implWrite(Unknown Source) >> at sun.nio.cs.StreamEncoder.write(Unknown Source) >> at java.io.OutputStreamWriter.write(Unknown Source) >> at java.io.BufferedWriter.flushBuffer(Unknown Source) >> at java.io.PrintStream.write(Unknown Source) >> at java.io.PrintStream.print(Unknown Source) >> at >> com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P >> rintStreamSocketLog.java:169) >> at >> com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P >> rintStreamSocketLog.java:175) >> at >> com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.logEr >> ror(PrintStreamSocketLog.java:71) >> at >> com.metamatrix.common.comm.platform.socket.client.SocketServerInstanc >> eImpl$1.run(SocketServerInstanceImpl.java:578) >> at java.lang.Thread.run(Unknown Source) >> Mar 13, 2010 3:52:09 PM org.apache.solr.update.DirectUpdateHandler2 >> rollback >> INFO: end_rollback >> >> Thanks, >> Barani >> >> JavaGuy84 wrote: >> > >> > Hi, >> > >> > I am using CachedsqlEntityProcessor in my DIH dataconfig to reduce the >> > number of queries executed against the database , >> > >> > <Entity1 query="select * from x" processor= >> "CachedsqlEntityProcessor"/> >> > <Entity2 query="select * from y" processor= "CachedsqlEntityProcessor" >> > cachekey="id" cachelookup="x.id"/> >> > >> > >> > I having more than 2 million rows returned for Entity 2 and around >> 300000 >> > rows returned for entity1. >> > >> > I am have set the heap size to 1 GB but even then I am always getting >> heap >> > out of size error. I am not sure how to flush the documents in buffer >> at >> > certain condition. I tried to enable Autocommit / reduced the >> > maxdocbuffersize but of no use.. Can someone let me know what is the >> best >> > way to overcome this issue? >> > >> > Thanks, >> > Barani >> > >> >> -- >> View this message in context: >> http://old.nabble.com/DIH---Out-of-Memory-error-when-using-CachedsqlEntityProcessor-tp27889623p27890751.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://old.nabble.com/DIH---Out-of-Memory-error-when-using-CachedsqlEntityProcessor-tp27889623p27891562.html Sent from the Solr - User mailing list archive at Nabble.com.
