Hi All, I triggered a /dataimport for first 100 rows from my database and while it's running issued another import request for rows 101-200.
In my log I see below exception; It seems multiple JDBC connections cannot be opened. Does this mean concurrency is not supported in DIH for JDBC datasources? Please share your thoughts on how to tackle concurrency in dataimport.. [Thread-15] ERROR org.apache.solr.handler.dataimport.JdbcDataSource - Ignoring Error when closing connection java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@1e820764 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries. at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924) at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:3314) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2477) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2809) at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:5165) at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5048) at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4654) at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1630) at org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:410) at org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:395) at org.apache.solr.handler.dataimport.DocBuilder.closeEntityProcessorWrappers(DocBuilder.java:284) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:273) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468) Thanks, Dileepa On Thu, Jan 30, 2014 at 4:13 PM, Dileepa Jayakody <dileepajayak...@gmail.com > wrote: > I would particularly like to know how DIH handles concurrency in JDBC > database connections during datamport.. > > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" > url="jdbc:mysql://localhost:3306/solrtest" user="usr1" password="123" > batchSize="1" /> > > Thanks, > Dileepa > > > On Thu, Jan 30, 2014 at 4:05 PM, Dileepa Jayakody < > dileepajayak...@gmail.com> wrote: > >> Hi All, >> >> Can I please know about how concurrency is handled in the DIH? >> What happens if multiple /dataimport requests are issued to the same >> Datasource? >> >> I'm doing some custom processing at the end of dataimport process as an >> EventListener configured in the data-config.xml as below. >> <document name="stanboldata" >> onImportEnd="com.solr.stanbol.processor.StanbolEventListener"> >> >> Will each DIH request create a new EventListener object? >> >> I'm copying some field values from my custom processor configured in the >> /dataimport request handler to a static Map in my StanbolEventListener >> class. >> I need to figure out how to handle concurrency when data is copied to my >> EvenetListener object to perform the rest of my update process. >> >> Thanks, >> Dileepa >> > >
