Re: Delta import throws java heap space exception
Hi Furkan, sure, this is my data-config.xml: dataConfig document entity name=item pk=id dataSource=store_db onError=skip query=SELECT IT.* FROM item AS IT JOIN order AS ORD ON IT.order_id=ORD.id WHERE (IT.status=1 AND ORD.status=1) deltaQuery=SELECT IT.* FROM item IT, order ORD, customer CUST WHERE IT.order_id1=ORD.id AND ORD.customer_id=CUST.id AND (IT.last_modified_date gt; DATE_SUB('${dataimporter.last_index_time}', INTERVAL 1 MINUTE) OR ORD.last_modified_date gt; DATE_SUB('${dataimporter.last_index_time}', INTERVAL 1 MINUTE) OR CUST.last_modified_date gt; DATE_SUB('${dataimporter.last_index_time}', INTERVAL 1 MINUTE)) deltaImportQuery=SELECT * FROM item WHERE id='${ dataimporter.delta.id}' deletedPkQuery=SELECT IT.id FROM item AS IT JOIN order AS ORD ON IT.order_id=ORD.id WHERE (IT.status!=1 OR ORD.status!=1) AND (IT.last_modified_date gt; DATE_SUB('${dataimporter.last_index_time}', INTERVAL 1 MINUTE) OR ORD.last_modified_date gt; DATE_SUB('${dataimporter.last_index_time}', INTERVAL 1 MINUTE)) field column=id name=id/ ... entity name=product dataSource=store_db query=SELECT PR.name, PR.description FROM product PR, item IT WHERE IT.product_id = PR.id AND IT.id='${item.id}' field column=name name=name/ ... /entity ... /entity /document /dataConfig Currently I have 2.1 Million of activities. Thanks a lot, Richard 2014-03-12 19:16 GMT-04:00 Furkan KAMACI furkankam...@gmail.com: Hi; Could you send your data-config.xml? Thanks; Furkan KAMACI 2014-03-13 1:01 GMT+02:00 Richard Marquina Lopez richard.marqu...@gmail.com : Hi Ahmet, Thank you for your response, currently I have the next configuration for JVM: -XX:+PrintGCDetails-XX:-UseParallelGC-XX:SurvivorRatio=8-XX:NewRatio=2 -XX:+HeapDumpOnOutOfMemoryError-XX:PermSize=128m-XX:MaxPermSize=256m -Xms1024m-Xmx2048m I have 3.67 GB of physical RAM and 2GB is asigned to JVM (-Xmx2048m) 2014-03-12 17:32 GMT-04:00 Ahmet Arslan iori...@yahoo.com: Hi Richard, How much ram do you assign to java heap? Try increasing it to 1 gb for example. Please see : https://wiki.apache.org/solr/ShawnHeisey Ahmet On Wednesday, March 12, 2014 10:53 PM, Richard Marquina Lopez richard.marqu...@gmail.com wrote: Hi, I have some problems when execute the delta import with 2 million of rows from mysql database: java.lang.OutOfMemoryError: Java heap space at java.nio.HeapCharBuffer.init(HeapCharBuffer.java:57) at java.nio.CharBuffer.allocate(CharBuffer.java:331) at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777) at java.nio.charset.Charset.decode(Charset.java:810) at com.mysql.jdbc.StringUtils.toString(StringUtils.java:2010) at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:820) at com.mysql.jdbc.BufferRow.getString(BufferRow.java:541) at com.mysql.jdbc.ResultSetImpl.getStringInternal( ResultSetImpl.java:5812) at com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5689) at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4986) at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:5175) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator.getARow(JdbcDataSource.java:315) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator.access$700(JdbcDataSource.java:254) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator$1.next(JdbcDataSource.java:294) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator$1.next(JdbcDataSource.java:286) at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext( EntityProcessorBase.java:117) at org.apache.solr.handler.dataimport.SqlEntityProcessor. nextModifiedRowKey(SqlEntityProcessor.java:86) at org.apache.solr.handler.dataimport.EntityProcessorWrapper. nextModifiedRowKey(EntityProcessorWrapper.java:267) at org.apache.solr.handler.dataimport.DocBuilder. collectDelta(DocBuilder.java:781) at org.apache.solr.handler.dataimport.DocBuilder.doDelta( DocBuilder.java:338) at org.apache.solr.handler.dataimport.DocBuilder.execute( DocBuilder.java:223) at org.apache.solr.handler.dataimport.DataImporter. doDeltaImport(DataImporter.java:440) at org.apache.solr.handler.dataimport.DataImporter. runCmd(DataImporter.java:478) at org.apache.solr.handler.dataimport.DataImporter$1.run( DataImporter.java:457) -- java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@47a034e7 is still active
Delta import throws java heap space exception
Hi, I have some problems when execute the delta import with 2 million of rows from mysql database: java.lang.OutOfMemoryError: Java heap space at java.nio.HeapCharBuffer.init(HeapCharBuffer.java:57) at java.nio.CharBuffer.allocate(CharBuffer.java:331) at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777) at java.nio.charset.Charset.decode(Charset.java:810) at com.mysql.jdbc.StringUtils.toString(StringUtils.java:2010) at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:820) at com.mysql.jdbc.BufferRow.getString(BufferRow.java:541) at com.mysql.jdbc.ResultSetImpl.getStringInternal( ResultSetImpl.java:5812) at com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5689) at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4986) at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:5175) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator.getARow(JdbcDataSource.java:315) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator.access$700(JdbcDataSource.java:254) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator$1.next(JdbcDataSource.java:294) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator$1.next(JdbcDataSource.java:286) at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext( EntityProcessorBase.java:117) at org.apache.solr.handler.dataimport.SqlEntityProcessor. nextModifiedRowKey(SqlEntityProcessor.java:86) at org.apache.solr.handler.dataimport.EntityProcessorWrapper. nextModifiedRowKey(EntityProcessorWrapper.java:267) at org.apache.solr.handler.dataimport.DocBuilder. collectDelta(DocBuilder.java:781) at org.apache.solr.handler.dataimport.DocBuilder.doDelta( DocBuilder.java:338) at org.apache.solr.handler.dataimport.DocBuilder.execute( DocBuilder.java:223) at org.apache.solr.handler.dataimport.DataImporter. doDeltaImport(DataImporter.java:440) at org.apache.solr.handler.dataimport.DataImporter. runCmd(DataImporter.java:478) at org.apache.solr.handler.dataimport.DataImporter$1.run( DataImporter.java:457) -- java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@47a034e7 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries. at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924) at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingDa ta(MysqlIO.java:3361) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828) at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks( ConnectionImpl.java:5204) at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5087) at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4690) at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1649) at org.apache.solr.handler.dataimport.JdbcDataSource. closeConnection(JdbcDataSource.java:436) at org.apache.solr.handler.dataimport.JdbcDataSource. close(JdbcDataSource.java:421) at org.apache.solr.handler.dataimport.DocBuilder. closeEntityProcessorWrappers(DocBuilder.java:288) at org.apache.solr.handler.dataimport.DocBuilder.execute( DocBuilder.java:277) at org.apache.solr.handler.dataimport.DataImporter. doDeltaImport(DataImporter.java:440) at org.apache.solr.handler.dataimport.DataImporter. runCmd(DataImporter.java:478) at org.apache.solr.handler.dataimport.DataImporter$1.run( DataImporter.java:457) Currently I have the batchSize parameter stetted to -1 Configuration: - SOLR 4.4 - Centos 5.5 - 2GB RAM - 1 Procesosr Does someone have the same error? Could someone help me, please? Thank you, Richard
Re: Delta import throws java heap space exception
Hi Ahmet, Thank you for your response, currently I have the next configuration for JVM: -XX:+PrintGCDetails-XX:-UseParallelGC-XX:SurvivorRatio=8-XX:NewRatio=2 -XX:+HeapDumpOnOutOfMemoryError-XX:PermSize=128m-XX:MaxPermSize=256m -Xms1024m-Xmx2048m I have 3.67 GB of physical RAM and 2GB is asigned to JVM (-Xmx2048m) 2014-03-12 17:32 GMT-04:00 Ahmet Arslan iori...@yahoo.com: Hi Richard, How much ram do you assign to java heap? Try increasing it to 1 gb for example. Please see : https://wiki.apache.org/solr/ShawnHeisey Ahmet On Wednesday, March 12, 2014 10:53 PM, Richard Marquina Lopez richard.marqu...@gmail.com wrote: Hi, I have some problems when execute the delta import with 2 million of rows from mysql database: java.lang.OutOfMemoryError: Java heap space at java.nio.HeapCharBuffer.init(HeapCharBuffer.java:57) at java.nio.CharBuffer.allocate(CharBuffer.java:331) at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777) at java.nio.charset.Charset.decode(Charset.java:810) at com.mysql.jdbc.StringUtils.toString(StringUtils.java:2010) at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:820) at com.mysql.jdbc.BufferRow.getString(BufferRow.java:541) at com.mysql.jdbc.ResultSetImpl.getStringInternal( ResultSetImpl.java:5812) at com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5689) at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4986) at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:5175) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator.getARow(JdbcDataSource.java:315) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator.access$700(JdbcDataSource.java:254) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator$1.next(JdbcDataSource.java:294) at org.apache.solr.handler.dataimport.JdbcDataSource$ ResultSetIterator$1.next(JdbcDataSource.java:286) at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext( EntityProcessorBase.java:117) at org.apache.solr.handler.dataimport.SqlEntityProcessor. nextModifiedRowKey(SqlEntityProcessor.java:86) at org.apache.solr.handler.dataimport.EntityProcessorWrapper. nextModifiedRowKey(EntityProcessorWrapper.java:267) at org.apache.solr.handler.dataimport.DocBuilder. collectDelta(DocBuilder.java:781) at org.apache.solr.handler.dataimport.DocBuilder.doDelta( DocBuilder.java:338) at org.apache.solr.handler.dataimport.DocBuilder.execute( DocBuilder.java:223) at org.apache.solr.handler.dataimport.DataImporter. doDeltaImport(DataImporter.java:440) at org.apache.solr.handler.dataimport.DataImporter. runCmd(DataImporter.java:478) at org.apache.solr.handler.dataimport.DataImporter$1.run( DataImporter.java:457) -- java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@47a034e7 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries. at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924) at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingDa ta(MysqlIO.java:3361) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828) at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks( ConnectionImpl.java:5204) at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5087) at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4690) at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1649) at org.apache.solr.handler.dataimport.JdbcDataSource. closeConnection(JdbcDataSource.java:436) at org.apache.solr.handler.dataimport.JdbcDataSource. close(JdbcDataSource.java:421) at org.apache.solr.handler.dataimport.DocBuilder. closeEntityProcessorWrappers(DocBuilder.java:288) at org.apache.solr.handler.dataimport.DocBuilder.execute( DocBuilder.java:277) at org.apache.solr.handler.dataimport.DataImporter. doDeltaImport(DataImporter.java:440) at org.apache.solr.handler.dataimport.DataImporter. runCmd(DataImporter.java:478) at org.apache.solr.handler.dataimport.DataImporter$1.run( DataImporter.java:457) Currently I have the batchSize parameter stetted to -1 Configuration: - SOLR 4.4 - Centos 5.5 - 2GB RAM - 1 Procesosr Does someone have the same