Ok, fix has been checked in. Karl
On Wed, Nov 28, 2012 at 3:19 AM, Karl Wright <daddy...@gmail.com> wrote: > The ticket is CONNECTORS-571. > > Karl > > On Wed, Nov 28, 2012 at 3:12 AM, Karl Wright <daddy...@gmail.com> wrote: >> Hi Shigeki, >> >> This confirms my theory that our MySQL driver is not detecting all >> cases where MySQL gives up on a transaction. We need to correct this, >> but in order to do that we need the SQL error code that MySQL throws >> in this case: >> >> Caused by: java.sql.SQLException: Lock wait timeout exceeded; try >> restarting transaction >> >> It looks like somebody actually posted the SQL error code that MYSQL >> sends out with this online: >> >> ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction >> >> Are you able to build ManifoldCF? I will check in a fix to trunk for >> this problem shortly; it would be great if you could try it out. >> >> Thanks, >> Karl >> >> On Wed, Nov 28, 2012 at 2:30 AM, Shigeki Kobayashi >> <shigeki.kobayas...@g.softbank.co.jp> wrote: >>> Hi Karl, >>> >>> >>> Here is a log of Database Exception that is occurred while crawling Web. >>> This time, socket timeout exception did not happen so it might be a >>> different matter. >>> Even though the job status remain "Running", it seems that MCF stopped >>> crawling (The job was not aborted). >>> -------------------------------- >>> ERROR 2012-11-22 19:36:28,593 (Worker thread '16') - Worker thread aborting >>> and restarting due to database connection reset: Database exception: >>> Exception doing query: Lock wait timeout exceeded; try restarting >>> transaction >>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database >>> exception: Exception doing query: Lock wait timeout exceeded; try restarting >>> transaction >>> at >>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681) >>> at >>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709) >>> at >>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394) >>> at >>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144) >>> at >>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186) >>> at >>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performModification(DBInterfaceMySQL.java:678) >>> at >>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performUpdate(DBInterfaceMySQL.java:275) >>> at >>> org.apache.manifoldcf.core.database.BaseTable.performUpdate(BaseTable.java:80) >>> at >>> org.apache.manifoldcf.crawler.jobs.HopCount.markForDelete(HopCount.java:1426) >>> at >>> org.apache.manifoldcf.crawler.jobs.HopCount.doDeleteInvalidation(HopCount.java:1356) >>> at >>> org.apache.manifoldcf.crawler.jobs.HopCount.doFinish(HopCount.java:1057) >>> at >>> org.apache.manifoldcf.crawler.jobs.HopCount.finishParents(HopCount.java:389) >>> at >>> org.apache.manifoldcf.crawler.jobs.JobManager.finishDocuments(JobManager.java:4309) >>> at >>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:557) >>> Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting >>> transaction >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541) >>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002) >>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163) >>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624) >>> at >>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127) >>> at >>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2427) >>> at >>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2345) >>> at >>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2330) >>> at >>> org.apache.manifoldcf.core.database.Database.execute(Database.java:840) >>> at >>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641) >>> >>> -------------------------------- >>> >>> >>> Here is a log of Database Exception that is occurred while crawling files >>> using Windows shares connection: >>> >>> >>> -------------------------------- >>> 2012/11/22 23:39:28 ERROR (Job start thread) - Job start thread aborting and >>> restarting due to database connection reset: Database exception: Exception >>> doing query: Lock wait timeout exceeded; try restarting transaction >>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database >>> exception: Exception doing query: Lock wait timeout exceeded; try restarting >>> transaction >>> at >>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681) >>> at >>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709) >>> at >>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394) >>> at >>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144) >>> at >>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186) >>> at >>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:852) >>> at >>> org.apache.manifoldcf.crawler.jobs.JobManager.startJobs(JobManager.java:4711) >>> at >>> org.apache.manifoldcf.crawler.system.JobStartThread.run(JobStartThread.java:68) >>> Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting >>> transaction >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609) >>> at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1578) >>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1434) >>> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2925) >>> at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477) >>> at >>> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2631) >>> at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1800) >>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2221) >>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624) >>> at >>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127) >>> at >>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293) >>> at >>> org.apache.manifoldcf.core.database.Database.execute(Database.java:826) >>> at >>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641) >>> 2012/11/22 23:39:28 ERROR (Finisher thread) - Finisher thread aborting and >>> restarting due to database connection reset: Database exception: Exception >>> doing query: Lock wait timeout exceeded; try restarting transaction >>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database >>> exception: Exception doing query: Lock wait timeout exceeded; try restarting >>> transaction >>> at >>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681) >>> at >>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709) >>> at >>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394) >>> at >>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144) >>> at >>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186) >>> at >>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:852) >>> at >>> org.apache.manifoldcf.crawler.jobs.JobManager.finishJobs(JobManager.java:6469) >>> at >>> org.apache.manifoldcf.crawler.system.FinisherThread.run(FinisherThread.java:64) >>> Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting >>> transaction >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609) >>> at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1578) >>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1434) >>> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2925) >>> at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477) >>> at >>> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2631) >>> at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1800) >>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2221) >>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624) >>> at >>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127) >>> at >>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293) >>> at >>> org.apache.manifoldcf.core.database.Database.execute(Database.java:826) >>> at >>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641) >>> 2012/11/22 23:39:30 ERROR (Worker thread '253') - Worker thread aborting and >>> restarting due to database connection reset: Database exception: Exception >>> doing query: Lock wait timeout exceeded; try restarting transaction >>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database >>> exception: Exception doing query: Lock wait timeout exceeded; try restarting >>> transaction >>> at >>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681) >>> at >>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709) >>> at >>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394) >>> at >>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144) >>> at >>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186) >>> at >>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performModification(DBInterfaceMySQL.java:678) >>> at >>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performUpdate(DBInterfaceMySQL.java:275) >>> at >>> org.apache.manifoldcf.core.database.BaseTable.performUpdate(BaseTable.java:80) >>> at >>> org.apache.manifoldcf.crawler.jobs.JobQueue.updateCompletedRecord(JobQueue.java:722) >>> at >>> org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentCompletedMultiple(JobManager.java:2435) >>> at >>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:765) >>> Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting >>> transaction >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541) >>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002) >>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163) >>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624) >>> at >>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127) >>> at >>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2427) >>> at >>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2345) >>> at >>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2330) >>> at >>> org.apache.manifoldcf.core.database.Database.execute(Database.java:840) >>> at >>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641) >>> -------------------------------- >>> Regards, >>> >>> Shigeki >>> >>> >>> 2012/11/27 Karl Wright <daddy...@gmail.com> >>>> >>>> Hi Shigeki, >>>> >>>> Deadlocks are a fact of life in a very multithreaded application. >>>> They are supposed to be caught by ManifoldCF, and the transactions >>>> retried. I can believe, though, that MySQL might set different >>>> sqlexception status codes for different kinds of deadlock - if instead >>>> of a sqlexception with a deadlock code, MySQL sometimes just drops the >>>> JDBC connection, that might explain the problem. >>>> >>>> Can you refresh my memory and please send the ManifoldCF log part that >>>> includes the socket timeout exception? I can then see if it is coming >>>> from the same place. >>>> >>>> Karl >>>> >>>> On Tue, Nov 27, 2012 at 12:50 AM, Shigeki Kobayashi >>>> <shigeki.kobayas...@g.softbank.co.jp> wrote: >>>> > Hi Karl, >>>> > >>>> > According to INNODB STATUS in MySQL, while crawling web, the following >>>> > DEADLOCK occurred. >>>> > A few minutes later, database exception occurred in MCF. >>>> > So do you think probably this DEADLOCK could cause the exception? >>>> > >>>> > I do not know the error code from MySQL yet, but maybe I could obtain it >>>> > if >>>> > you could let me >>>> > know what code in what file should be added in order to output the error >>>> > code into manifold.log >>>> > >>>> > >>>> > >>>> > ------------------------------------------------------------------------------ >>>> > INNODB STATUS: >>>> > ===================================== >>>> > 121127 11:17:49 INNODB MONITOR OUTPUT >>>> > ===================================== >>>> > Per second averages calculated from the last 60 seconds >>>> > ----------------- >>>> > BACKGROUND THREAD >>>> > ----------------- >>>> > srv_master_thread loops: 401163 1_second, 401162 sleeps, 40049 >>>> > 10_second, >>>> > 674 background, 674 flush >>>> > srv_master_thread log flush and writes: 401182 >>>> > ---------- >>>> > SEMAPHORES >>>> > ---------- >>>> > OS WAIT ARRAY INFO: reservation count 7319, signal count 6842 >>>> > Mutex spin waits 3702, rounds 111120, OS waits 3626 >>>> > RW-shared spins 2189, rounds 63516, OS waits 1767 >>>> > RW-excl spins 255, rounds 57147, OS waits 1897 >>>> > Spin rounds per wait: 30.02 mutex, 29.02 RW-shared, 224.11 RW-excl >>>> > ------------------------ >>>> > LATEST DETECTED DEADLOCK >>>> > ------------------------ >>>> > 121122 19:31:55 >>>> > *** (1) TRANSACTION: >>>> > TRANSACTION 3021A0, ACTIVE 32 sec starting index read >>>> > mysql tables in use 1, locked 1 >>>> > LOCK WAIT 64 lock struct(s), heap size 14776, 110 row lock(s), undo log >>>> > entries 51 >>>> > MySQL thread id 24, OS thread handle 0x7ff8ffe06700, query id 41385 >>>> > 10.249.23.9 manifoldcf Sending data >>>> > SELECT parentidhash,linktype,distance FROM hopcount WHERE >>>> > jobid=1351139121625 AND parentidhash IN >>>> > >>>> > ('A0ED08F9D45547FF54B72869FE5E7C3C5B0E910A','F5E2F6C6B43FB5D030C4F0AE8E22AD07536475A8','C0856A1AFF55F7BB20BCAE317E18F588EEFB806D','019253D99FCB265A20A3CFF11D0443937FE2D4D0','21A979F9BB9120F747B0B605EDABA71EB364A584','A8B5B7245D0810584B764470B42CFDF71C33A7E5','6FE272988943D3BD64E285951A1A6739011FC15E','1E1CA954A3E31BFC28FFE1BE70757408341CDB6A','8DAE8B4734A30FE2D346EEBD1CCC3A16468F7B7B','028CD3E7FF7F493E3EC3980FF303DB05DC42404E','924E0608A5C4505C9272A69B8C1F82C7B883A11F','13F6402C96E0979EF5F17338DFF96BD9912125D9','43174E34AA07C34237D622A43A82AFE3825C3870','32892282A6866BD181BDA0BA85801192370C84F3','0311197289655163E1452E90D43A5D96D9A4E751','178C8BE84AEDC9F362CE3A2CC2702F6C2CD9CBA1','7CF15B193B3BDA097BEB437272FC5E413B86B63D') >>>> > AND linktype IN ('link','redirect') >>>> > *** (1) WAITING FOR THIS LOCK TO BE GRANTED: >>>> > RECORD LOCKS space id 0 page no 2449 n bits 192 index `PRIMARY` of table >>>> > `manifoldcf`.`hopcount` trx id 3021A0 lock mode S locks rec but not gap >>>> > waiting >>>> > Record lock, heap no 28 PHYSICAL RECORD: n_fields 8; compact format; >>>> > info >>>> > bits 0 >>>> > 0: len 8; hex 8000013b261c6d8a; asc ;& m ;; >>>> > 1: len 6; hex 0000002f4e47; asc /NG;; >>>> > 2: len 7; hex 900000021b06ec; asc ;; >>>> > 3: len 1; hex 4e; asc N;; >>>> > 4: len 30; hex >>>> > 384441453842343733344133304645324433343645454244314343433341; asc >>>> > 8DAE8B4734A30FE2D346EEBD1CCC3A; (total 40 bytes); >>>> > 5: len 8; hex 8000000000000001; asc ;; >>>> > 6: len 8; hex 8000013a962ad9d9; asc : * ;; >>>> > 7: len 4; hex 6c696e6b; asc link;; >>>> > >>>> > *** (2) TRANSACTION: >>>> > TRANSACTION 302208, ACTIVE 3 sec fetching rows >>>> > mysql tables in use 3, locked 3 >>>> > 1436 lock struct(s), heap size 145848, 122906 row lock(s) >>>> > MySQL thread id 39, OS thread handle 0x7ff8ffa37700, query id 40699 >>>> > 10.249.23.9 manifoldcf preparing >>>> > UPDATE hopcount SET deathmark='D',distance=-1 WHERE id IN(SELECT ownerid >>>> > FROM hopdeletedeps t0 WHERE t0.jobid=1351139121625 AND >>>> > t0.childidhash='D573BDC6D59C7A7CC2862646322F69EA5574C36D' AND >>>> > EXISTS(SELECT >>>> > 'x' FROM intrinsiclink t1 WHERE t1.jobid=t0.jobid AND >>>> > t1.linktype=t0.linktype AND t1.parentidhash=t0.parentidhash AND >>>> > t1.childidhash=t0.childidhash AND t1.isnew='B')) >>>> > *** (2) HOLDS THE LOCK(S): >>>> > RECORD LOCKS space id 0 page no 2449 n bits 192 index `PRIMARY` of table >>>> > `manifoldcf`.`hopcount` trx id 302208 lock_mode X >>>> > Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info >>>> > bits 0 >>>> > 0: len 8; hex 73757072656d756d; asc supremum;; >>>> > >>>> > ... >>>> > ... >>>> > ... >>>> > >>>> > *** WE ROLL BACK TRANSACTION (1) >>>> > ------------ >>>> > TRANSACTIONS >>>> > ------------ >>>> > Trx id counter 38375F >>>> > Purge done for trx's n:o < 3024F7 undo n:o < 0 >>>> > History list length 652 >>>> > LIST OF TRANSACTIONS FOR EACH SESSION: >>>> > ---TRANSACTION 0, not started >>>> > MySQL thread id 110, OS thread handle 0x7ff914113700, query id 1436936 >>>> > localhost root >>>> > SHOW ENGINE INNODB STATUS >>>> > ---TRANSACTION 0, not started >>>> > MySQL thread id 106, OS thread handle 0x7ff9035b3700, query id 1435785 >>>> > localhost root >>>> > ---TRANSACTION 38375E, not started >>>> > MySQL thread id 99, OS thread handle 0x7ff8ff72b700, query id 1436934 >>>> > 10.249.23.9 manifoldcf >>>> > -------- >>>> > ... >>>> > >>>> > ------------------------------------------------------------------------------ >>>> > >>>> > >>>> > Likewise, file crawling using Windows shares faced similar matter. >>>> > DEADLOCK >>>> > occured in MySQL and >>>> > Database Exception occured in MCF as well: >>>> > >>>> > >>>> > ------------------------------------------------------------------------------ >>>> > >>>> > | InnoDB | | >>>> > ===================================== >>>> > 121126 16:05:21 INNODB MONITOR OUTPUT >>>> > ===================================== >>>> > Per second averages calculated from the last 48 seconds >>>> > ----------------- >>>> > BACKGROUND THREAD >>>> > ----------------- >>>> > srv_master_thread loops: 327427 1_second, 327300 sleeps, 32438 >>>> > 10_second, >>>> > 3544 background, 3544 flush >>>> > srv_master_thread log flush and writes: 327670 >>>> > ---------- >>>> > SEMAPHORES >>>> > ---------- >>>> > OS WAIT ARRAY INFO: reservation count 1808090, signal count 2140762 >>>> > Mutex spin waits 18194682, rounds 103331992, OS waits 842070 >>>> > RW-shared spins 1311114, rounds 25796436, OS waits 457767 >>>> > RW-excl spins 577964, rounds 15904805, OS waits 333210 >>>> > Spin rounds per wait: 5.68 mutex, 19.68 RW-shared, 27.52 RW-excl >>>> > ------------------------ >>>> > LATEST DETECTED DEADLOCK >>>> > ------------------------ >>>> > 121122 23:38:46 >>>> > *** (1) TRANSACTION: >>>> > TRANSACTION 674749, ACTIVE 7 sec inserting >>>> > mysql tables in use 1, locked 1 >>>> > LOCK WAIT 4 lock struct(s), heap size 1248, 3 row lock(s), undo log >>>> > entries >>>> > 1 >>>> > MySQL thread id 99, OS thread handle 0x7f7d4a356700, query id 23942404 >>>> > localhost 127.0.0.1 manifoldcf update >>>> > INSERT INTO jobqueue >>>> > >>>> > (docpriority,id,priorityset,docid,status,dochash,checktime,checkaction,jobid) >>>> > VALUES >>>> > >>>> > (13.830866056523654,1353595119848,1353595119385,'smb://xxx/xxx','P','88517951DB2E0666151E7B5308C9FDCB16F062AD',0,'R',1353575409046) >>>> > *** (1) WAITING FOR THIS LOCK TO BE GRANTED: >>>> > RECORD LOCKS space id 0 page no 221894 n bits 208 index `I1352346865065` >>>> > of >>>> > table `manifoldcf`.`jobqueue` trx id 674749 lock_mode X locks gap before >>>> > rec >>>> > insert intention waiting >>>> > Record lock, heap no 134 PHYSICAL RECORD: n_fields 3; compact format; >>>> > info >>>> > bits 0 >>>> > 0: len 30; hex >>>> > 383835313838444433453134444134354242384531383433424330393444; asc >>>> > 885188DD3E14DA45BB8E1843BC094D; (total 40 bytes); >>>> > 1: len 8; hex 8000013b2761a596; asc ;'a ;; >>>> > 2: len 8; hex 8000013b287bd5c9; asc ;({ ;; >>>> > >>>> > *** (2) TRANSACTION: >>>> > TRANSACTION 6740DF, ACTIVE 9 sec fetching rows >>>> > mysql tables in use 5, locked 5 >>>> > 23571 lock struct(s), heap size 2439608, 1058037 row lock(s) >>>> > MySQL thread id 45, OS thread handle 0x7f7d21231700, query id 23937374 >>>> > localhost 127.0.0.1 manifoldcf Sending data >>>> > SELECT >>>> > >>>> > t0.id,t0.jobid,t0.dochash,t0.docid,t0.status,t0.failtime,t0.failcount,t0.priorityset >>>> > FROM jobqueue t0 WHERE t0.status IN ('P','G') AND t0.checkaction='R' AND >>>> > t0.checktime<=1353595117855 AND EXISTS(SELECT 'x' FROM jobs t1 WHERE >>>> > t1.status IN ('A','a') AND t1.id=t0.jobid AND t1.priority=5) AND NOT >>>> > EXISTS(SELECT 'x' FROM jobqueue t2 WHERE t2.dochash=t0.dochash AND >>>> > t2.status >>>> > IN ('A','F','a','f','D','d') AND t2.jobid!=t0.jobid) AND NOT >>>> > EXISTS(SELECT >>>> > 'x' FROM prereqevents t3,events t4 WHERE t0.id=t3.owner AND >>>> > t3.eventname=t4.name) ORDER BY t0.docpriority ASC,t0.status >>>> > ASC,t0.checkaction ASC,t0.checktime ASC LIMIT 1200 >>>> > *** (2) HOLDS THE LOCK(S): >>>> > RECORD LOCKS space id 0 page no 221894 n bits 208 index `I1352346865065` >>>> > of >>>> > table `manifoldcf`.`jobqueue` trx id 6740DF lock mode S locks gap before >>>> > rec >>>> > Record lock, heap no 8 PHYSICAL RECORD: n_fields 3; compact format; info >>>> > bits 0 >>>> > 0: len 30; hex >>>> > 383834464239393738383632333242323331353041343031303337424444; asc >>>> > 884FB997886232B23150A401037BDD; (total 40 bytes); >>>> > 1: len 8; hex 8000013b2761a596; asc ;'a ;; >>>> > 2: len 8; hex 8000013b27c4823b; asc ;' ;;; >>>> > >>>> > ... >>>> > ... >>>> > ... >>>> > >>>> > *** WE ROLL BACK TRANSACTION (1) >>>> > ------------ >>>> > TRANSACTIONS >>>> > ------------ >>>> > Trx id counter 6ACDF6 >>>> > Purge done for trx's n:o < 6752D1 undo n:o < 0 >>>> > History list length 485 >>>> > LIST OF TRANSACTIONS FOR EACH SESSION: >>>> > ---TRANSACTION 0, not started >>>> > MySQL thread id 5505, OS thread handle 0x7f7d210ec700, query id 25071245 >>>> > localhost root >>>> > SHOW ENGINE INNODB STATUS >>>> > ---TRANSACTION 6ACDF5, not started >>>> > MySQL thread id 99, OS thread handle 0x7f7d4a356700, query id 25071244 >>>> > localhost 127.0.0.1 manifoldcf >>>> > ... >>>> > >>>> > ------------------------------------------------------------------------------ >>>> > >>>> > >>>> > >>>> > >>>> > Regards, >>>> > >>>> > Shigeki >>>> > >>>> > >>>> > 2012/10/19 Shigeki Kobayashi <shigeki.kobayas...@g.softbank.co.jp> >>>> >> >>>> >> Due to the error, I had to downgrade to a lower version so I haven't >>>> >> found >>>> >> the MySQL error code yet. >>>> >> >>>> >> I installed MCF1.0 in a different environment where crawlable contents >>>> >> are >>>> >> different from the above environment. >>>> >> I could not reproduce the Database exception but socket timeout >>>> >> occurred >>>> >> In the same environment, I ran MCF0.6 and it completed crawling without >>>> >> socket timeout. >>>> >> Like you said, socket timeout seems to be a different problem from the >>>> >> Database exception . >>>> >> >>>> >> 2012/10/18 Karl Wright <daddy...@gmail.com> >>>> >>> >>>> >>> So, what was the resolution of this problem? Any news? >>>> >>> Karl >>>> >>> >>>> >>> On Thu, Oct 11, 2012 at 2:28 AM, Karl Wright <daddy...@gmail.com> >>>> >>> wrote: >>>> >>> > The only change is that the MySQL driver now performs ANALYZE >>>> >>> > operations on the fly in order to keep the database operating at >>>> >>> > high >>>> >>> > efficiency. This is CONNECTORS-510. It is possible that, on a >>>> >>> > large >>>> >>> > database table, these operations will cause others to wait long >>>> >>> > enough >>>> >>> > so that their timeout is exceeded. Such an event does not take >>>> >>> > place >>>> >>> > while the load tests run, however. If you want to turn off the >>>> >>> > analyze operation, you can do that by setting a per-table property >>>> >>> > to >>>> >>> > override the analyze default of 10000 operations: >>>> >>> > >>>> >>> > analyzeThreshold = >>>> >>> > >>>> >>> > >>>> >>> > ManifoldCF.getIntProperty("org.apache.manifold.db.mysql.analyze."+tableName,10000); >>>> >>> > >>>> >>> > The table in question is "jobqueue". If you set this value to >>>> >>> > something like 1000000000 and you still see MySQL timeouts, then >>>> >>> > this >>>> >>> > new code is not the problem. And, like I said, the best solution is >>>> >>> > to recognize the error and retry, but first I would need the error >>>> >>> > code. Adding an appropriate output of sqlState around line 123 of >>>> >>> > >>>> >>> > >>>> >>> > framework/core/src/main/java/org/apache/manifoldcf/core/database/DBInterfaceMySQL.java >>>> >>> > would allow us to see what code to catch, when it happened again. >>>> >>> > >>>> >>> > For the Web connector, the only modifications have been in regards >>>> >>> > to >>>> >>> > how it handles 500 errors, which now correctly code to avoid an >>>> >>> > IndexExceptionOutOfBounds exception. This has nothing to do with >>>> >>> > socket exceptions, which are caused for external reasons only. >>>> >>> > >>>> >>> > Karl >>>> >>> > >>>> >>> > >>>> >>> > On Wed, Oct 10, 2012 at 10:32 PM, Shigeki Kobayashi >>>> >>> > <shigeki.kobayas...@g.softbank.co.jp> wrote: >>>> >>> >> Hi Karl, >>>> >>> >> >>>> >>> >> >>>> >>> >> I was comparing version 1.0 with old trunk based on version 0.6 >>>> >>> >> implementing >>>> >>> >> CONNECTORS-501( >>>> >>> >> Medium-scale web crawl with hopcount-based filtering fails to find >>>> >>> >> correct >>>> >>> >> number of documents). >>>> >>> >> >>>> >>> >> Running each version with the same MySQL setting and the same >>>> >>> >> throttling, >>>> >>> >> somehow the version 1.0 hangs with the error. >>>> >>> >> Since the old trunk completes crawling, I wonder if something has >>>> >>> >> changed. >>>> >>> >> >>>> >>> >> Just to make sure I will recheck if there are any wrong settings in >>>> >>> >> MCF. >>>> >>> >> >>>> >>> >> Thanks. >>>> >>> >> >>>> >>> >> Regards, >>>> >>> >> >>>> >>> >> Shigeki >>>> >>> >> >>>> >>> >> 2012/10/10 Karl Wright <daddy...@gmail.com> >>>> >>> >>> >>>> >>> >>> Hi Shigeki, >>>> >>> >>> >>>> >>> >>> The socket timeout exception is only a warning. It means that >>>> >>> >>> some >>>> >>> >>> site you are crawling did not accept a socket connection within >>>> >>> >>> the >>>> >>> >>> allowed time (5 minutes I think). The Web Connector will retry >>>> >>> >>> the >>>> >>> >>> connection a few times, and if it is still rejected, it will >>>> >>> >>> eventually give up on that page. One thing you want to check, >>>> >>> >>> though, >>>> >>> >>> is that you are using proper throttling, because if you aren't >>>> >>> >>> then >>>> >>> >>> one cause of this problem is that the webmaster of the site you >>>> >>> >>> are >>>> >>> >>> trying to crawl may have blocked you from accessing it. >>>> >>> >>> >>>> >>> >>> The database exception is more problematic. It means that MySQL >>>> >>> >>> thinks it took too long for a specific transaction to complete, >>>> >>> >>> and >>>> >>> >>> the database aborted the transaction due to a timeout. There are >>>> >>> >>> two >>>> >>> >>> ways of dealing with this issue. One way is to modify your MySQL >>>> >>> >>> configuration to increase the transaction timeout value to some >>>> >>> >>> high >>>> >>> >>> number. The second way is to modify ManifoldCF to recognize the >>>> >>> >>> timeout error specifically, and cause a retry. But in order to do >>>> >>> >>> the >>>> >>> >>> latter, I would need to know what SQL error code MySQL returns for >>>> >>> >>> this situation, which will mean we either need to look it up (if >>>> >>> >>> we >>>> >>> >>> can), or modify a ManifoldCF instance to log it when this problem >>>> >>> >>> occurs. >>>> >>> >>> >>>> >>> >>> Please let me know how you would like to proceed. >>>> >>> >>> >>>> >>> >>> Karl >>>> >>> >>> >>>> >>> >>> On Wed, Oct 10, 2012 at 3:51 AM, Shigeki Kobayashi >>>> >>> >>> <shigeki.kobayas...@g.softbank.co.jp> wrote: >>>> >>> >>> > >>>> >>> >>> > Hi >>>> >>> >>> > >>>> >>> >>> > I am having a trouble with crawling web using MCF1.0. >>>> >>> >>> > I run MCF with MySQL 5.5 and Tomcat 6.0. >>>> >>> >>> > It should keep crawling contents, but MCF prints the following >>>> >>> >>> > Database >>>> >>> >>> > exception log, then hangs. >>>> >>> >>> > After DB Exception, Socket Time Exception occurs. >>>> >>> >>> > >>>> >>> >>> > Anyone has faced this problem? >>>> >>> >>> > >>>> >>> >>> > --Database Exception log: >>>> >>> >>> > >>>> >>> >>> > ERROR 2012-10-10 16:11:05,787 (Worker thread '42') - Worker >>>> >>> >>> > thread >>>> >>> >>> > aborting >>>> >>> >>> > and restarting due to database connection reset: Database >>>> >>> >>> > exception: >>>> >>> >>> > Exception doing query: Lock wait timeout exceeded; try >>>> >>> >>> > restarting >>>> >>> >>> > transaction >>>> >>> >>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException: >>>> >>> >>> > Database >>>> >>> >>> > exception: Exception doing query: Lock wait timeout exceeded; >>>> >>> >>> > try >>>> >>> >>> > restarting >>>> >>> >>> > transaction >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:852) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.addDocumentReference(WorkerThread.java:1487) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityLinkHandler.noteDiscoveredLink(WebcrawlerConnector.java:6049) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessAcivityHTMLHandler.noteAHREF(WebcrawlerConnector.java:6159) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.LinkParseState.noteNonscriptTag(LinkParseState.java:44) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.FormParseState.noteNonscriptTag(FormParseState.java:52) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState.noteTag(ScriptParseState.java:50) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.BasicParseState.dealWithCharacter(BasicParseState.java:225) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.handleHTML(WebcrawlerConnector.java:7047) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.extractLinks(WebcrawlerConnector.java:6011) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.processDocuments(WebcrawlerConnector.java:1282) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:551) >>>> >>> >>> > Caused by: java.sql.SQLException: Lock wait timeout exceeded; >>>> >>> >>> > try >>>> >>> >>> > restarting >>>> >>> >>> > transaction >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073) >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609) >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541) >>>> >>> >>> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002) >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163) >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database.execute(Database.java:826) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641) >>>> >>> >>> > ERROR 2012-10-10 16:11:06,799 (Worker thread '9') - Worker >>>> >>> >>> > thread >>>> >>> >>> > aborting >>>> >>> >>> > and restarting due to database connection reset: Database >>>> >>> >>> > exception: >>>> >>> >>> > Exception doing query: Lock wait timeout exceeded; try >>>> >>> >>> > restarting >>>> >>> >>> > transaction >>>> >>> >>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException: >>>> >>> >>> > Database >>>> >>> >>> > exception: Exception doing query: Lock wait timeout exceeded; >>>> >>> >>> > try >>>> >>> >>> > restarting >>>> >>> >>> > transaction >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:852) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.flush(WorkerThread.java:1863) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:554) >>>> >>> >>> > Caused by: java.sql.SQLException: Lock wait timeout exceeded; >>>> >>> >>> > try >>>> >>> >>> > restarting >>>> >>> >>> > transaction >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073) >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609) >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541) >>>> >>> >>> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002) >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163) >>>> >>> >>> > at >>>> >>> >>> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database.execute(Database.java:826) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641) >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > ---- Socket Timeout: >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > DEBUG 2012-10-10 16:16:27,256 (Worker thread '49') - Socket >>>> >>> >>> > timeout >>>> >>> >>> > exception trying to close connection: Read timed out >>>> >>> >>> > java.net.SocketTimeoutException: Read timed out >>>> >>> >>> > at java.net.SocketInputStream.socketRead0(Native Method) >>>> >>> >>> > at >>>> >>> >>> > java.net.SocketInputStream.read(SocketInputStream.java:129) >>>> >>> >>> > at >>>> >>> >>> > java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >>>> >>> >>> > at >>>> >>> >>> > java.io.BufferedInputStream.read1(BufferedInputStream.java:258) >>>> >>> >>> > at >>>> >>> >>> > java.io.BufferedInputStream.read(BufferedInputStream.java:317) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.read(Unknown >>>> >>> >>> > Source) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.read(Unknown >>>> >>> >>> > Source) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.commons.httpclient.ChunkedInputStream.exhaustInputStream(Unknown >>>> >>> >>> > Source) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.close(Unknown >>>> >>> >>> > Source) >>>> >>> >>> > at >>>> >>> >>> > java.io.FilterInputStream.close(FilterInputStream.java:155) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.commons.httpclient.AutoCloseInputStream.notifyWatcher(Unknown >>>> >>> >>> > Source) >>>> >>> >>> > at >>>> >>> >>> > org.apache.commons.httpclient.AutoCloseInputStream.close(Unknown >>>> >>> >>> > Source) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.close(ThrottledFetcher.java:2082) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:176) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:745) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321) >>>> >>> >>> > INFO 2012-10-10 16:16:27,273 (Worker thread '49') - WEB: FETCH >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > URL|http://xxxxxx/...|1349852786744+600514|-104|4125|org.apache.manifoldcf.core.interfaces.ManifoldCFException| >>>> >>> >>> > Interrupted: Socket timeout: Read timed out >>>> >>> >>> > DEBUG 2012-10-10 16:16:27,273 (Worker thread '49') - WEB: Fetch >>>> >>> >>> > exception >>>> >>> >>> > for 'http://xxxxxx/...' >>>> >>> >>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException: >>>> >>> >>> > Interrupted: >>>> >>> >>> > Socket timeout: Read timed out >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.noteInterrupted(ThrottledFetcher.java:1818) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:797) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321) >>>> >>> >>> > Caused by: >>>> >>> >>> > org.apache.manifoldcf.agents.interfaces.ServiceInterruption: >>>> >>> >>> > Socket timeout: Read timed out >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:745) >>>> >>> >>> > ... 1 more >>>> >>> >>> > Caused by: java.net.SocketTimeoutException: Read timed out >>>> >>> >>> > at java.net.SocketInputStream.socketRead0(Native Method) >>>> >>> >>> > at >>>> >>> >>> > java.net.SocketInputStream.read(SocketInputStream.java:129) >>>> >>> >>> > at >>>> >>> >>> > java.io.BufferedInputStream.read1(BufferedInputStream.java:256) >>>> >>> >>> > at >>>> >>> >>> > java.io.BufferedInputStream.read(BufferedInputStream.java:317) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.read(Unknown >>>> >>> >>> > Source) >>>> >>> >>> > at >>>> >>> >>> > java.io.FilterInputStream.read(FilterInputStream.java:116) >>>> >>> >>> > at >>>> >>> >>> > org.apache.commons.httpclient.AutoCloseInputStream.read(Unknown >>>> >>> >>> > Source) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2012) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:1976) >>>> >>> >>> > at >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:95) >>>> >>> >>> > ... 2 more >>>> >>> >>> > WARN 2012-10-10 16:16:27,274 (Worker thread '49') - Pre-ingest >>>> >>> >>> > service >>>> >>> >>> > interruption reported for job 1349774325961 connection 'WEB': >>>> >>> >>> > Socket >>>> >>> >>> > timeout: Read timed out >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > >>>> >>> >>> > Regards, >>>> >>> >>> > >>>> >>> >>> > Shigeki >>>> >>> >> >>>> >>> >> >>>> >>> >> >>>> >>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> > >>>> > >>>> > >>> >>> >>> >>>