Re: Deduplication patch not working in nightly build

Mark Miller Fri, 09 Jan 2009 05:50:27 -0800

I can't imagine why dedupe would have anything to do with this, otherthan what was said, it perhaps is taking a bit longer to get a documentto the db, and it times out (maybe a long signature calculation?). Haveyou tried changing your MySql settings to allow for a longer timeout?(sorry, I'm not to up to date on what you have tried).

Also, are you using autocommit during the import? If so, you might tryturning it off for the full import.


- Mark

Marc Sturlese wrote:

Hey there,
I am stack in this problem sine 3 days ago and no idea how to sort it.

I am using the nighlty from a week ago, mysql and this driver and url:
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/my_db"

I can use deduplication patch with indexs of 200.000 docs and no problem.
When I try a full-import with a db of 1.500.000 it stops indexing at doc
number 15.000 aprox showing me the error posted above.
Once I get the exception, i restart tomcat and start a delta-import... this
time everything works fine!
I need to avoid this error in the full import, i have tryed:

url="jdbc:mysql://localhost/my_db?autoReconnect=true to sort it in case the
connection was closed due to long time until next doc was indexed, but
nothing changed... I keep having this:
Jan 9, 2009 1:38:18 PM org.apache.solr.handler.dataimport.JdbcDataSource
logError

WARNING: Error reading datacom.mysql.jdbc.CommunicationsException: Communications link failure due tounderlying exception:** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
        at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:279)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
        at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
        at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
        at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
        at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
        at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
        at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)


** END NESTED EXCEPTION **



Last packet sent to the server was 206097 ms ago.
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
        at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:279)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
        at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
        at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
        at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
        at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
        at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
        at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)
Jan 9, 2009 1:38:18 PM org.apache.solr.handler.dataimport.JdbcDataSource
logError
WARNING: Exception while closing result set
com.mysql.jdbc.CommunicationsException: Communications link failure due to

underlying exception:** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
        at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
        at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
        at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:295)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:287)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
        at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
        at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
        at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
        at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
        at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
        at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)


** END NESTED EXCEPTION **



Last packet sent to the server was 206097 ms ago.
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
        at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
        at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
        at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:295)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:287)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
        at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
        at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
        at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
        at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
        at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
        at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)


Has anyone experienced this problem? Any advice what could i do...
Thanks


Marc Sturlese wrote:

Thanks I will have a look to my JdbcDataSource. Anyway it's weird because
using the 1.3 release I don't have that problem...

Shalin Shekhar Mangar wrote:

Yes, initially I figured that we are accidentally re-using a closed data
source. But Noble has pinned it right. I guess you can try looking into
your
JDBC driver's documentation for a setting which increases the connection
alive-ness.

On Mon, Jan 5, 2009 at 5:29 PM, Noble Paul നോബിള്‍ नोब्ळ् <
noble.p...@gmail.com> wrote:

I guess the indexing of a doc is taking too long (may be because of
the de-dup patch) and the resultset gets closed automaticallly (timed
out)
--Noble

On Mon, Jan 5, 2009 at 5:14 PM, Marc Sturlese <marc.sturl...@gmail.com>
wrote:

Donig this fix I get the same error :(

I am going to try to set up the last nigthly build... let's see if I

have

better luck.

The thing is it stop indexing at the doc num 150.000 aprox... and give

me

that mysql exception error... Without DeDuplication patch I can index

milion docs without problems...

I am pretty lost with this... :(


Shalin Shekhar Mangar wrote:

Yes I meant the 05/01/2008 build. The fix is a one line change

Add the following as the last line of DataConfig.Entity.clearCache()
dataSrc = null;



On Mon, Jan 5, 2009 at 4:22 PM, Marc Sturlese
<marc.sturl...@gmail.com>wrote:

Shalin you mean I should test the 05/01/2008 nighlty? maybe with

this
one

works? If the fix you did is not really big can u tell me where in

the

source is and what is it for? (I have been debuging and tracing a

lot
the

dataimporthandler source and I I would like to know what the

imporovement

is
about if it is not a problem...)

Thanks!


Shalin Shekhar Mangar wrote:

Marc, I've just committed a fix which may have caused the bug. Can

you

use

svn trunk (or the next nightly build) and confirm?

On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള്‍ नोब्ळ् <
noble.p...@gmail.com> wrote:

looks like a bug w/ DIH with the recent fixes.
--Noble

On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese

<marc.sturl...@gmail.com>

wrote:

Hey there,
I was using the Deduplication patch with Solr 1.3 release and

everything
was

working perfectly. Now I upgraded to a nigthly build (20th

december)

to

be

able to use new facet algorithm and other stuff and

DeDuplication
is

not

working any more. I have followed exactly the same steps to

apply

the

patch

to the source code. I am geting this error:

WARNING: Error reading data
com.mysql.jdbc.CommunicationsException: Communications link

failure

due

to

underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException
       at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
       at

com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)

at

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)

at

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)

       at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
       at

com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)

at

com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)

       at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
       at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)

at

org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)

at

org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)

at

org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)

at

org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)

at

org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)

at

org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)

at

org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)

at

org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)

** END NESTED EXCEPTION **
Last packet sent to the server was 202481 ms ago.
       at

com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)

at

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)

at

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)

       at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
       at

com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)

at

com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)

       at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
       at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)

at

org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)

at

org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)

at

org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)

at

org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)

at

org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)

at

org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)

at

org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)

at

org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)

Jan 5, 2009 10:06:16 AM

org.apache.solr.handler.dataimport.JdbcDataSource

logError
WARNING: Exception while closing result set
com.mysql.jdbc.CommunicationsException: Communications link

failure

due

to

underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException
       at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
       at

com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)

at

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)

at

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)

       at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
       at

com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)

at

com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)

at

com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)

at

com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)

       at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
       at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)

at

org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)

at

org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)

at

org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)

at

org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)

at

org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)

at

org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)

at

org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)

at

org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)

** END NESTED EXCEPTION **

Last packet sent to the server was 202482 ms ago.
       at

com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)

at

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)

at

com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)

       at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
       at

com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)

at

com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)

at

com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)

at

com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)

       at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
       at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)

at

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)

at

org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)

at

org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)

at

org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)

at

org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)

at

org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)

at

org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)

at

org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)

at

org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)

Mark do you know if I should to some extra hack to run

DeDuplication

patch

in the new solr version (nightly)? Has something changed in the

source

that

make it not work or it's simply that i am doing something

wrong?

Thanks in advanced
--
View this message in context:

http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html

Sent from the Solr - User mailing list archive at Nabble.com.


--
--Noble Paul


--
Regards,
Shalin Shekhar Mangar.

--
View this message in context:

http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html

Sent from the Solr - User mailing list archive at Nabble.com.

--
Regards,
Shalin Shekhar Mangar.

--
View this message in context:

http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21289643.html

Sent from the Solr - User mailing list archive at Nabble.com.


--
--Noble Paul


--
Regards,
Shalin Shekhar Mangar.

Re: Deduplication patch not working in nightly build

Reply via email to