[
https://issues.apache.org/jira/browse/SOLR-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254855#comment-13254855
]
Claudio R commented on SOLR-3360:
---------------------------------
Hi James,
About the version 3.5.0, I got unstable behavior with 10 threads. In first
full-import, I got successful import:
<lst name="statusMessages">
<str name="Total Requests made to DataSource">0</str>
<str name="Total Rows Fetched">1000</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2012-04-16 14:12:08</str>
<str name="">Indexing completed. Added/Updated: 1000 documents. Deleted 0
documents.</str>
<str name="Committed">2012-04-16 14:13:21</str>
<str name="Optimized">2012-04-16 14:13:21</str>
<str name="Total Documents Processed">1000</str>
<str name="Time taken ">0:1:12.875</str>
</lst>
But, in second, third full-import I got Indexing failed. Rolled back all
changes.
<lst name="statusMessages">
<str name="Time Elapsed">0:0:6.906</str>
<str name="Total Requests made to DataSource">0</str>
<str name="Total Rows Fetched">12</str>
<str name="Total Documents Processed">11</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2012-04-16 14:15:38</str>
<str name="">Indexing failed. Rolled back all changes.</str>
<str name="Rolledback">2012-04-16 14:15:43</str>
</lst>
At catalina.out, I got:
SEVERE: Full Import failed:java.lang.RuntimeException: Error in multi-threaded
import
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:265)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:187)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:359)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:427)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
Unable to execute query: select categoria, sub_categoria from filtro where
indice_id_indice = '257346'
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper.nextRow(ThreadedEntityProcessorWrapper.java:84)
at
org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.runAThread(DocBuilder.java:446)
at
org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.run(DocBuilder.java:399)
at
org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.runAThread(DocBuilder.java:466)
at
org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.access$000(DocBuilder.java:353)
at
org.apache.solr.handler.dataimport.DocBuilder$EntityRunner$1.run(DocBuilder.java:406)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: socket closed
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1368)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1355)
at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1532)
at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3274)
at
com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:4433)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:784)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:685)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4026)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:185)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:160)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.execute(SQLServerStatement.java:658)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
... 13 more
In version 3.6.0 I did not get unstable behavior as obtained in version 3.5.0
with 10 threads.
In version 3.6.0 I tried without transformers:
<lst name="statusMessages">
<str name="Total Requests made to DataSource">0</str>
<str name="Total Rows Fetched">10000</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2012-04-16 14:30:39</str>
<str name="">Indexing completed. Added/Updated: 10000 documents. Deleted 0
documents.</str>
<str name="Committed">2012-04-16 14:39:45</str>
<str name="Total Documents Processed">10000</str>
<str name="Time taken">0:9:5.719</str>
</lst>
In version 3.6.0, with just parent entity:
<lst name="statusMessages">
<str name="Total Requests made to DataSource">0</str>
<str name="Total Rows Fetched">10000</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2012-04-16 14:42:49</str>
<str name="">Indexing completed. Added/Updated: 10000 documents. Deleted 0
documents.</str>
<str name="Committed">2012-04-16 14:49:05</str>
<str name="Total Documents Processed">10000</str>
<str name="Time taken">0:6:16.0</str>
</lst>
It's weird obtain 10000 documents processed. I only have 1000 records in the
database.
Thanks.
> Problem with DataImportHandler multi-threaded
> ---------------------------------------------
>
> Key: SOLR-3360
> URL: https://issues.apache.org/jira/browse/SOLR-3360
> Project: Solr
> Issue Type: Bug
> Affects Versions: 3.6
> Environment: Solr 3.6.0, Apache Tomcat 6.0.20, jdk1.6.0_15, Windows XP
> Reporter: Claudio R
>
> Hi,
> If I use dataimport with 1 thread, I got:
> <lst name="statusMessages">
> <str name="Total Requests made to DataSource">5001</str>
> <str name="Total Rows Fetched">1000</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2012-04-16 11:21:57</str>
> <str name="">Indexing completed. Added/Updated: 1000 documents. Deleted 0
> documents.</str>
> <str name="Committed">2012-04-16 11:23:19</str>
> <str name="Total Documents Processed">1000</str>
> <str name="Time taken">0:1:22.390</str>
> </lst>
> If I use datamport with 10 threads, I got:
> <lst name="statusMessages">
> <str name="Total Requests made to DataSource">0</str>
> <str name="Total Rows Fetched">10000</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2012-04-16 11:31:43</str>
> <str name="">Indexing completed. Added/Updated: 10000 documents. Deleted 0
> documents.</str>
> <str name="Committed">2012-04-16 11:41:50</str>
> <str name="Total Documents Processed">10000</str>
> <str name="Time taken">0:10:7.586</str>
> </lst>
> The configuration of 10 threads consumed 10 times longer than the
> configuration with 1 thread.
> I have 1000 records in the database.
> My db-data-config.xml is shown below:
> <?xml version="1.0" encoding="UTF-8" ?>
> <dataConfig>
> <dataSource driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
> url="jdbc:sqlserver://200.XXX.XXX.XXX:1433;databaseName=test" user="user"
> password="pass"/>
> <document>
> <entity name="indice" rootEntity="true" threads="10"
> transformer="RegexTransformer,TemplateTransformer" query="select top 1000
> i.id_indice, i.a, i.b from indice i where i.status = 'I'"
> deltaImportQuery="i.id_indice, i.a, i.b from indice i where id_indice in
> ('${dataimporter.delta.id_indice}')" deltaQuery="select id_indice from indice
> where status='I' and data_hora_modificacao >= convert(datetime,
> '${dataimporter.last_index_time}', 120)" deletedPkQuery="select id_indice
> from indice where status='D' and data_hora_modificacao >= convert(datetime,
> '${dataimporter.last_index_time}', 120)">
> <field column="id_indice" name="id_indice" />
> <field column="a" name="a" />
> <field column="b" name="b" />
> <entity name="filtro"
> transformer="RegexTransformer,TemplateTransformer" query="select categoria,
> sub_categoria from filtro where indice_id_indice = '${indice.id_indice}'">
> <field name="filtro_categoria" column="categoria" />
> <field name="filtro_sub_categoria" column="sub_categoria" />
> <field name="nv_sub_categoria" column="nv_sub_categoria"
> template="${filtro.categoria}|${filtro.sub_categoria}" />
> </entity>
> <entity name="pagina_relacionada" query="select url from
> pagina_relacionada where indice_id_indice = '${indice.id_indice}'">
> <field name="pagina_relacionada_url" column="url" />
> </entity>
> <entity name="veja_mais" query="select chamada, url from
> veja_mais where indice_id_indice = '${indice.id_indice}'">
> <field name="veja_mais_chamada" column="chamada" />
> <field name="veja_mais_url" column="url" />
> </entity>
> <entity name="video" query="select url from video where
> indice_id_indice = '${indice.id_indice}'">
> <field name="video_url" column="url" />
> </entity>
> <entity name="galeria" query="select url from galeria where
> indice_id_indice = '${indice.id_indice}'">
> <field name="galeria_url" column="url" />
> </entity>
> </entity>
> </document>
> </dataConfig>
> Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
