Re: Delta import throws java heap space exception

Richard Marquina Lopez Thu, 13 Mar 2014 11:55:29 -0700

Hi Furkan,

sure, this is my data-config.xml:


  <dataConfig>
  <document>
    <entity name="item" pk="id" dataSource="store_db" onError="skip"
        query="SELECT IT.* FROM item AS IT JOIN order AS ORD ON
IT.order_id=ORD.id WHERE (IT.status=1 AND ORD.status=1)"
        deltaQuery="SELECT IT.* FROM item IT, order ORD, customer CUST
WHERE IT.order_id1=ORD.id AND ORD.customer_id=CUST.id AND
(IT.last_modified_date &gt; DATE_SUB('${dataimporter.last_index_time}',
INTERVAL 1 MINUTE) OR ORD.last_modified_date &gt;
DATE_SUB('${dataimporter.last_index_time}', INTERVAL 1 MINUTE) OR
CUST.last_modified_date &gt; DATE_SUB('${dataimporter.last_index_time}',
INTERVAL 1 MINUTE))"
        deltaImportQuery="SELECT * FROM item WHERE id='${
dataimporter.delta.id}'"
        deletedPkQuery="SELECT IT.id FROM item AS IT JOIN order AS ORD ON
IT.order_id=ORD.id WHERE (IT.status!=1 OR ORD.status!=1) AND
(IT.last_modified_date &gt; DATE_SUB('${dataimporter.last_index_time}',
INTERVAL 1 MINUTE) OR ORD.last_modified_date &gt;
DATE_SUB('${dataimporter.last_index_time}', INTERVAL 1 MINUTE))">

      <field column="id" name="id"/>
      ...

      <entity name="product" dataSource="store_db" query="SELECT PR.name,
PR.description FROM product PR, item IT WHERE IT.product_id = PR.id AND
IT.id='${item.id}'" >
        <field column="name" name="name"/>
        ...
      </entity>
      ...
    </entity>
  </document>
</dataConfig>

Currently I have 2.1 Million of activities.

Thanks a lot,
Richard


2014-03-12 19:16 GMT-04:00 Furkan KAMACI <furkankam...@gmail.com>:

> Hi;
>
> Could you send your data-config.xml?
>
> Thanks;
> Furkan KAMACI
>
>
> 2014-03-13 1:01 GMT+02:00 Richard Marquina Lopez <
> richard.marqu...@gmail.com
> >:
>
> > Hi Ahmet,
> >
> > Thank you for your response, currently I have the next configuration for
> > JVM:
> > -XX:+PrintGCDetails-XX:-UseParallelGC-XX:SurvivorRatio=8-XX:NewRatio=2
> > -XX:+HeapDumpOnOutOfMemoryError-XX:PermSize=128m-XX:MaxPermSize=256m
> > -Xms1024m-Xmx2048m
> > I have 3.67 GB of physical RAM and 2GB is asigned to JVM (-Xmx2048m)
> >
> >
> > 2014-03-12 17:32 GMT-04:00 Ahmet Arslan <iori...@yahoo.com>:
> >
> > > Hi Richard,
> > >
> > > How much ram do you assign to java heap? Try increasing it to 1 gb for
> > > example.
> > > Please see : https://wiki.apache.org/solr/ShawnHeisey
> > >
> > > Ahmet
> > >
> > >
> > >
> > > On Wednesday, March 12, 2014 10:53 PM, Richard Marquina Lopez <
> > > richard.marqu...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I have some problems when execute the delta import with 2 million of
> rows
> > > from mysql database:
> > >
> > > java.lang.OutOfMemoryError: Java heap space
> > >         at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
> > >         at java.nio.CharBuffer.allocate(CharBuffer.java:331)
> > >         at
> > java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777)
> > >         at java.nio.charset.Charset.decode(Charset.java:810)
> > >         at com.mysql.jdbc.StringUtils.toString(StringUtils.java:2010)
> > >         at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:820)
> > >         at com.mysql.jdbc.BufferRow.getString(BufferRow.java:541)
> > >         at com.mysql.jdbc.ResultSetImpl.getStringInternal(
> > > ResultSetImpl.java:5812)
> > >         at
> > com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5689)
> > >         at
> > com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4986)
> > >         at
> > com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:5175)
> > >         at org.apache.solr.handler.dataimport.JdbcDataSource$
> > > ResultSetIterator.getARow(JdbcDataSource.java:315)
> > >         at org.apache.solr.handler.dataimport.JdbcDataSource$
> > > ResultSetIterator.access$700(JdbcDataSource.java:254)
> > >         at org.apache.solr.handler.dataimport.JdbcDataSource$
> > > ResultSetIterator$1.next(JdbcDataSource.java:294)
> > >         at org.apache.solr.handler.dataimport.JdbcDataSource$
> > > ResultSetIterator$1.next(JdbcDataSource.java:286)
> > >         at
> > org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(
> > > EntityProcessorBase.java:117)
> > >         at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> > > nextModifiedRowKey(SqlEntityProcessor.java:86)
> > >         at org.apache.solr.handler.dataimport.EntityProcessorWrapper.
> > > nextModifiedRowKey(EntityProcessorWrapper.java:267)
> > >         at org.apache.solr.handler.dataimport.DocBuilder.
> > > collectDelta(DocBuilder.java:781)
> > >         at org.apache.solr.handler.dataimport.DocBuilder.doDelta(
> > > DocBuilder.java:338)
> > >         at org.apache.solr.handler.dataimport.DocBuilder.execute(
> > > DocBuilder.java:223)
> > >         at org.apache.solr.handler.dataimport.DataImporter.
> > > doDeltaImport(DataImporter.java:440)
> > >         at org.apache.solr.handler.dataimport.DataImporter.
> > > runCmd(DataImporter.java:478)
> > >         at org.apache.solr.handler.dataimport.DataImporter$1.run(
> > > DataImporter.java:457)
> > > ------------------------------------------------------------
> > >
> >
> --------------------------------------------------------------------------
> > >
> > > java.sql.SQLException: Streaming result set
> > > com.mysql.jdbc.RowDataDynamic@47a034e7
> > > is still active.
> > > No statements may be issued when any streaming result sets are open and
> > in
> > > use on a given connection.
> > > Ensure that you have called .close() on any active streaming result
> sets
> > > before attempting more queries.
> > >         at
> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
> > >         at
> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
> > >         at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingDa
> > > ta(MysqlIO.java:3361)
> > >         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
> > >         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> > >         at
> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
> > >         at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(
> > > ConnectionImpl.java:5204)
> > >         at
> > com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5087)
> > >         at
> > > com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4690)
> > >         at
> com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1649)
> > >         at org.apache.solr.handler.dataimport.JdbcDataSource.
> > > closeConnection(JdbcDataSource.java:436)
> > >         at org.apache.solr.handler.dataimport.JdbcDataSource.
> > > close(JdbcDataSource.java:421)
> > >         at org.apache.solr.handler.dataimport.DocBuilder.
> > > closeEntityProcessorWrappers(DocBuilder.java:288)
> > >         at org.apache.solr.handler.dataimport.DocBuilder.execute(
> > > DocBuilder.java:277)
> > >         at org.apache.solr.handler.dataimport.DataImporter.
> > > doDeltaImport(DataImporter.java:440)
> > >         at org.apache.solr.handler.dataimport.DataImporter.
> > > runCmd(DataImporter.java:478)
> > >         at org.apache.solr.handler.dataimport.DataImporter$1.run(
> > > DataImporter.java:457)
> > >
> > > Currently I have the batchSize parameter stetted to -1
> > >
> > > Configuration:
> > > - SOLR 4.4
> > > - Centos 5.5
> > > - 2GB RAM
> > > - 1 Procesosr
> > >
> > > Does someone have the same error?
> > > Could someone help me, please?
> > >
> > > Thank you,
> > > Richard
> > >
>

Re: Delta import throws java heap space exception

Reply via email to