[
https://issues.apache.org/jira/browse/CONNECTORS-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720982#comment-16720982
]
Karl Wright commented on CONNECTORS-1562:
-----------------------------------------
Attached the "reduced" step with query logging. Analysis will take some time.
The entire startup log chunk is here (and it contains the seeding part, which
is what we're interested in):
{code}
DEBUG 2018-12-14T01:07:42,367 (Startup thread) - Requested query: [UPDATE
jobqueue SET needpriorityprocessid=NULL,needpriority=? WHERE (jobid=? AND
status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,368 (Thread-688) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,368 (Thread-688) - Done actual query (0ms): [SET
SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,369 (Thread-689) - Actual query: [UPDATE jobqueue SET
needpriorityprocessid=NULL,needpriority=? WHERE (jobid=? AND status=? OR
jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,369 (Thread-689) - Parameter 0: 'T'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) - Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) - Parameter 2: 'P'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) - Parameter 3: '1544121003866'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) - Parameter 4: 'G'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) - Done actual query (0ms): [UPDATE
jobqueue SET needpriorityprocessid=NULL,needpriority=? WHERE (jobid=? AND
status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,369 (Startup thread) - Beginning transaction of type 2
DEBUG 2018-12-14T01:07:42,369 (Startup thread) - Marking for delete for job
1544121003866 all hopcount document references from table jobqueue t99 matching
t99.status IN (?,?)
DEBUG 2018-12-14T01:07:42,370 (Startup thread) - Requested query: [UPDATE
hopcount SET distance=?,deathmark=? WHERE id IN(SELECT t0.ownerid FROM
hopdeletedeps t0,jobqueue t99,intrinsiclink t1 WHERE t0.jobid=? AND (t1.jobid=?
AND t1.parentidhash=t0.parentidhash AND t1.linktype=t0.linktype AND
t1.childidhash=t0.childidhash) AND (t99.jobid=? AND t99.dochash=t0.childidhash)
AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,370 (Thread-690) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,370 (Thread-690) - Done actual query (0ms): [SET
SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,371 (Thread-691) - Actual query: [UPDATE hopcount SET
distance=?,deathmark=? WHERE id IN(SELECT t0.ownerid FROM hopdeletedeps
t0,jobqueue t99,intrinsiclink t1 WHERE t0.jobid=? AND (t1.jobid=? AND
t1.parentidhash=t0.parentidhash AND t1.linktype=t0.linktype AND
t1.childidhash=t0.childidhash) AND (t99.jobid=? AND t99.dochash=t0.childidhash)
AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,371 (Thread-691) - Parameter 0: '-1'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) - Parameter 1: 'D'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) - Parameter 3: '1544121003866'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) - Parameter 4: '1544121003866'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) - Parameter 5: 'P'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) - Parameter 6: 'H'
DEBUG 2018-12-14T01:07:42,389 (Thread-691) - Done actual query (18ms): [UPDATE
hopcount SET distance=?,deathmark=? WHERE id IN(SELECT t0.ownerid FROM
hopdeletedeps t0,jobqueue t99,intrinsiclink t1 WHERE t0.jobid=? AND (t1.jobid=?
AND t1.parentidhash=t0.parentidhash AND t1.linktype=t0.linktype AND
t1.childidhash=t0.childidhash) AND (t99.jobid=? AND t99.dochash=t0.childidhash)
AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,390 (Startup thread) - Done setting hopcount rows for
job 1544121003866 to initial distances
DEBUG 2018-12-14T01:07:42,390 (Startup thread) - Requested query: [DELETE FROM
intrinsiclink WHERE EXISTS(SELECT 'x' FROM jobqueue t99 WHERE (t99.jobid=? AND
t99.dochash=intrinsiclink.childidhash) AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,390 (Thread-692) - Actual query: [DELETE FROM
intrinsiclink WHERE EXISTS(SELECT 'x' FROM jobqueue t99 WHERE (t99.jobid=? AND
t99.dochash=intrinsiclink.childidhash) AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,391 (Thread-692) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,391 (Thread-692) - Parameter 1: 'P'
DEBUG 2018-12-14T01:07:42,391 (Thread-692) - Parameter 2: 'H'
DEBUG 2018-12-14T01:07:42,407 (Thread-692) - Done actual query (17ms): [DELETE
FROM intrinsiclink WHERE EXISTS(SELECT 'x' FROM jobqueue t99 WHERE (t99.jobid=?
AND t99.dochash=intrinsiclink.childidhash) AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,408 (Startup thread) - Requested query: [DELETE FROM
hopdeletedeps WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND
deathmark=?))]
DEBUG 2018-12-14T01:07:42,408 (Thread-693) - Actual query: [DELETE FROM
hopdeletedeps WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND
deathmark=?))]
DEBUG 2018-12-14T01:07:42,409 (Thread-693) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,409 (Thread-693) - Parameter 1: 'D'
DEBUG 2018-12-14T01:07:42,410 (Thread-693) - Done actual query (2ms): [DELETE
FROM hopdeletedeps WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND
deathmark=?))]
DEBUG 2018-12-14T01:07:42,410 (Startup thread) - Requested query: [UPDATE
hopcount SET deathmark=? WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,411 (Thread-694) - Actual query: [UPDATE hopcount SET
deathmark=? WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,411 (Thread-694) - Parameter 0: 'Q'
DEBUG 2018-12-14T01:07:42,411 (Thread-694) - Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,411 (Thread-694) - Parameter 2: 'D'
DEBUG 2018-12-14T01:07:42,411 (Thread-694) - Done actual query (0ms): [UPDATE
hopcount SET deathmark=? WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,411 (Startup thread) - Done queueing for deletion for
1544121003866
DEBUG 2018-12-14T01:07:42,411 (Startup thread) - Requested query: [DELETE FROM
prereqevents WHERE EXISTS(SELECT 'x' FROM jobqueue t0 WHERE
t0.id=prereqevents.owner AND (t0.jobid=? AND t0.status=? OR t0.jobid=? AND
t0.status=?))]
DEBUG 2018-12-14T01:07:42,412 (Thread-695) - Actual query: [DELETE FROM
prereqevents WHERE EXISTS(SELECT 'x' FROM jobqueue t0 WHERE
t0.id=prereqevents.owner AND (t0.jobid=? AND t0.status=? OR t0.jobid=? AND
t0.status=?))]
DEBUG 2018-12-14T01:07:42,412 (Thread-695) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,412 (Thread-695) - Parameter 1: 'P'
DEBUG 2018-12-14T01:07:42,412 (Thread-695) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,412 (Thread-695) - Parameter 3: 'H'
DEBUG 2018-12-14T01:07:42,419 (Thread-695) - Done actual query (7ms): [DELETE
FROM prereqevents WHERE EXISTS(SELECT 'x' FROM jobqueue t0 WHERE
t0.id=prereqevents.owner AND (t0.jobid=? AND t0.status=? OR t0.jobid=? AND
t0.status=?))]
DEBUG 2018-12-14T01:07:42,419 (Startup thread) - Requested query: [DELETE FROM
jobqueue WHERE (jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,420 (Thread-696) - Actual query: [DELETE FROM
jobqueue WHERE (jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,420 (Thread-696) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,420 (Thread-696) - Parameter 1: 'P'
DEBUG 2018-12-14T01:07:42,420 (Thread-696) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,420 (Thread-696) - Parameter 3: 'H'
DEBUG 2018-12-14T01:07:42,421 (Thread-696) - Done actual query (2ms): [DELETE
FROM jobqueue WHERE (jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,421 (Startup thread) - Requested query: [UPDATE
jobqueue SET failtime=NULL,checktime=?,failcount=NULL,checkaction=NULL,status=?
WHERE (jobid=? AND status=? OR jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Actual query: [UPDATE jobqueue SET
failtime=NULL,checktime=?,failcount=NULL,checkaction=NULL,status=? WHERE
(jobid=? AND status=? OR jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Parameter 0: '0'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Parameter 1: 'Z'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Parameter 3: 'G'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Parameter 4: '1544121003866'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Parameter 5: 'U'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Parameter 6: '1544121003866'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Parameter 7: 'C'
DEBUG 2018-12-14T01:07:42,423 (Thread-697) - Done actual query (1ms): [UPDATE
jobqueue SET failtime=NULL,checktime=?,failcount=NULL,checkaction=NULL,status=?
WHERE (jobid=? AND status=? OR jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,423 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,439 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,454 (Startup thread) - Beginning transaction of type 1
DEBUG 2018-12-14T01:07:42,454 (Startup thread) - Requested query: [SELECT
bincounter FROM docbins WHERE (connectorclass=? AND binname=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,454 (Thread-698) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,454 (Thread-698) - Done actual query (0ms): [SET
SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,455 (Thread-699) - Actual query: [SELECT bincounter
FROM docbins WHERE (connectorclass=? AND binname=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,455 (Thread-699) - Parameter 0:
'org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector'
DEBUG 2018-12-14T01:07:42,455 (Thread-699) - Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,455 (Thread-699) - Done actual query (0ms): [SELECT
bincounter FROM docbins WHERE (connectorclass=? AND binname=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,455 (Startup thread) - Requested query: [INSERT INTO
docbins (connectorclass,bincounter,binname) VALUES (?,?,?)]
DEBUG 2018-12-14T01:07:42,456 (Thread-700) - Actual query: [INSERT INTO docbins
(connectorclass,bincounter,binname) VALUES (?,?,?)]
DEBUG 2018-12-14T01:07:42,456 (Thread-700) - Parameter 0:
'org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector'
DEBUG 2018-12-14T01:07:42,456 (Thread-700) - Parameter 1: '5.0'
DEBUG 2018-12-14T01:07:42,456 (Thread-700) - Parameter 2: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,456 (Thread-700) - Done actual query (0ms): [INSERT
INTO docbins (connectorclass,bincounter,binname) VALUES (?,?,?)]
DEBUG 2018-12-14T01:07:42,456 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,457 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,457 (Startup thread) - Beginning transaction of type 2
DEBUG 2018-12-14T01:07:42,457 (Startup thread) - Requested query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,458 (Thread-701) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,458 (Thread-701) - Done actual query (0ms): [SET
SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,458 (Thread-702) - Actual query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,458 (Thread-702) - Parameter 0:
'00EFFBADCA2DC5EA9A03967A85DBCD87090B05B2'
DEBUG 2018-12-14T01:07:42,458 (Thread-702) - Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,459 (Thread-702) - Done actual query (1ms): [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,460 (Startup thread) - Requested query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,460 (Thread-703) - Actual query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,460 (Thread-703) - Parameter 0: '1544767286064'
DEBUG 2018-12-14T01:07:42,461 (Thread-703) - Done actual query (1ms): [DELETE
FROM prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,461 (Startup thread) - Requested query: [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Actual query: [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Parameter 0: '0.0'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Parameter 7: '1544767286064'
DEBUG 2018-12-14T01:07:42,462 (Thread-704) - Done actual query (1ms): [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,463 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,463 (Thread-705) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,463 (Thread-705) - Parameter 0: '1544767286064'
DEBUG 2018-12-14T01:07:42,463 (Thread-705) - Parameter 1:
':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,463 (Thread-705) - Done actual query (0ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,463 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,464 (Thread-706) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,464 (Thread-706) - Parameter 0: '1544767286064'
DEBUG 2018-12-14T01:07:42,464 (Thread-706) - Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,464 (Thread-706) - Done actual query (0ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,465 (Startup thread) - Requested query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,465 (Thread-707) - Actual query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,465 (Thread-707) - Parameter 0:
'1B901235B7D91C61B5F50851437A10527E97ECF3'
DEBUG 2018-12-14T01:07:42,465 (Thread-707) - Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,466 (Thread-707) - Done actual query (1ms): [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,466 (Startup thread) - Requested query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,466 (Thread-708) - Actual query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,466 (Thread-708) - Parameter 0: '1544767286063'
DEBUG 2018-12-14T01:07:42,467 (Thread-708) - Done actual query (1ms): [DELETE
FROM prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,467 (Startup thread) - Requested query: [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Actual query: [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Parameter 0: '0.6931471805599453'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Parameter 7: '1544767286063'
DEBUG 2018-12-14T01:07:42,468 (Thread-709) - Done actual query (1ms): [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,468 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,468 (Thread-710) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,468 (Thread-710) - Parameter 0: '1544767286063'
DEBUG 2018-12-14T01:07:42,468 (Thread-710) - Parameter 1:
':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,468 (Thread-710) - Done actual query (0ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,469 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,469 (Thread-711) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,469 (Thread-711) - Parameter 0: '1544767286063'
DEBUG 2018-12-14T01:07:42,469 (Thread-711) - Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,469 (Thread-711) - Done actual query (0ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,469 (Startup thread) - Requested query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,469 (Thread-712) - Actual query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,469 (Thread-712) - Parameter 0:
'3B08425D39C140EB4403190B7481C7A90A8435FF'
DEBUG 2018-12-14T01:07:42,469 (Thread-712) - Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,470 (Thread-712) - Done actual query (1ms): [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,470 (Startup thread) - Requested query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,471 (Thread-713) - Actual query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,471 (Thread-713) - Parameter 0: '1544767286062'
DEBUG 2018-12-14T01:07:42,471 (Thread-713) - Done actual query (0ms): [DELETE
FROM prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,471 (Startup thread) - Requested query: [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,471 (Thread-714) - Actual query: [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,471 (Thread-714) - Parameter 0: '1.0986122886681098'
DEBUG 2018-12-14T01:07:42,471 (Thread-714) - Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) - Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) - Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) - Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,472 (Thread-714) - Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) - Parameter 7: '1544767286062'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) - Done actual query (1ms): [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,472 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,473 (Thread-715) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,473 (Thread-715) - Parameter 0: '1544767286062'
DEBUG 2018-12-14T01:07:42,473 (Thread-715) - Parameter 1:
':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,473 (Thread-715) - Done actual query (0ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,473 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,474 (Thread-716) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,474 (Thread-716) - Parameter 0: '1544767286062'
DEBUG 2018-12-14T01:07:42,474 (Thread-716) - Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,474 (Thread-716) - Done actual query (0ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,474 (Startup thread) - Requested query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,475 (Thread-717) - Actual query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,475 (Thread-717) - Parameter 0:
'9DC7778DE94DBCD1BD2A7D7BAEFAB09919972A78'
DEBUG 2018-12-14T01:07:42,475 (Thread-717) - Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,476 (Thread-717) - Done actual query (1ms): [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,476 (Startup thread) - Requested query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,476 (Thread-718) - Actual query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,476 (Thread-718) - Parameter 0: '1544767286059'
DEBUG 2018-12-14T01:07:42,477 (Thread-718) - Done actual query (1ms): [DELETE
FROM prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,477 (Startup thread) - Requested query: [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Actual query: [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Parameter 0: '1.3862943611198906'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Parameter 7: '1544767286059'
DEBUG 2018-12-14T01:07:42,478 (Thread-719) - Done actual query (1ms): [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,478 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,478 (Thread-720) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,478 (Thread-720) - Parameter 0: '1544767286059'
DEBUG 2018-12-14T01:07:42,478 (Thread-720) - Parameter 1:
':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,479 (Thread-720) - Done actual query (1ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,479 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,479 (Thread-721) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,479 (Thread-721) - Parameter 0: '1544767286059'
DEBUG 2018-12-14T01:07:42,479 (Thread-721) - Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,480 (Thread-721) - Done actual query (1ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,480 (Startup thread) - Requested query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,480 (Thread-722) - Actual query: [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,480 (Thread-722) - Parameter 0:
'BCE78F106BCB6F912A5A10B55D182A53504AA6C2'
DEBUG 2018-12-14T01:07:42,480 (Thread-722) - Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,480 (Thread-722) - Done actual query (0ms): [SELECT
id,status,checktime FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,481 (Startup thread) - Requested query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,481 (Thread-723) - Actual query: [DELETE FROM
prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,481 (Thread-723) - Parameter 0: '1544767286058'
DEBUG 2018-12-14T01:07:42,481 (Thread-723) - Done actual query (0ms): [DELETE
FROM prereqevents WHERE owner=?]
DEBUG 2018-12-14T01:07:42,481 (Startup thread) - Requested query: [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Actual query: [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Parameter 0: '1.6094379124341003'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Parameter 7: '1544767286058'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Done actual query (0ms): [UPDATE
jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,482 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,483 (Thread-725) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,483 (Thread-725) - Parameter 0: '1544767286058'
DEBUG 2018-12-14T01:07:42,483 (Thread-725) - Parameter 1:
':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,483 (Thread-725) - Done actual query (0ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,483 (Startup thread) - Requested query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,484 (Thread-726) - Actual query: [INSERT INTO
prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,484 (Thread-726) - Parameter 0: '1544767286058'
DEBUG 2018-12-14T01:07:42,484 (Thread-726) - Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,484 (Thread-726) - Done actual query (0ms): [INSERT
INTO prereqevents (owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,484 (Startup thread) - Requested query: [SELECT
parentidhash FROM intrinsiclink WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=? OR jobid=? AND parentidhash=? AND linktype=? AND
childidhash=? OR jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR
jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=? AND
parentidhash=? AND linktype=? AND childidhash=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Actual query: [SELECT parentidhash
FROM intrinsiclink WHERE (jobid=? AND parentidhash=? AND linktype=? AND
childidhash=? OR jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR
jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=? AND
parentidhash=? AND linktype=? AND childidhash=? OR jobid=? AND parentidhash=?
AND linktype=? AND childidhash=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 1:
'00EFFBADCA2DC5EA9A03967A85DBCD87090B05B2'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 2: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 3: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 4: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 5:
'1B901235B7D91C61B5F50851437A10527E97ECF3'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 6: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 7: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 8: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 9:
'3B08425D39C140EB4403190B7481C7A90A8435FF'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 10: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 11: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 12: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 13:
'9DC7778DE94DBCD1BD2A7D7BAEFAB09919972A78'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 14: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 15: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 16: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 17:
'BCE78F106BCB6F912A5A10B55D182A53504AA6C2'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 18: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Parameter 19: ''
DEBUG 2018-12-14T01:07:42,486 (Thread-727) - Done actual query (1ms): [SELECT
parentidhash FROM intrinsiclink WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=? OR jobid=? AND parentidhash=? AND linktype=? AND
childidhash=? OR jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR
jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=? AND
parentidhash=? AND linktype=? AND childidhash=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,486 (Startup thread) - Requested query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Actual query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Parameter 0: ''
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Parameter 3:
'3B08425D39C140EB4403190B7481C7A90A8435FF'
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Parameter 4: ''
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Done actual query (1ms): [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,487 (Startup thread) - Requested query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Actual query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Parameter 0: ''
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Parameter 3:
'1B901235B7D91C61B5F50851437A10527E97ECF3'
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Parameter 4: ''
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Done actual query (0ms): [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,488 (Startup thread) - Requested query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Actual query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Parameter 0: ''
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Parameter 3:
'BCE78F106BCB6F912A5A10B55D182A53504AA6C2'
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Parameter 4: ''
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Done actual query (0ms): [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,489 (Startup thread) - Requested query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Actual query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Parameter 0: ''
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Parameter 3:
'9DC7778DE94DBCD1BD2A7D7BAEFAB09919972A78'
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Parameter 4: ''
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Done actual query (0ms): [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Startup thread) - Requested query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Thread-732) - Actual query: [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Thread-732) - Parameter 0: ''
DEBUG 2018-12-14T01:07:42,490 (Thread-732) - Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,490 (Thread-732) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,490 (Thread-732) - Parameter 3:
'00EFFBADCA2DC5EA9A03967A85DBCD87090B05B2'
DEBUG 2018-12-14T01:07:42,491 (Thread-732) - Parameter 4: ''
DEBUG 2018-12-14T01:07:42,491 (Thread-732) - Parameter 5: ''
DEBUG 2018-12-14T01:07:42,491 (Thread-732) - Done actual query (1ms): [UPDATE
intrinsiclink SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND
linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,491 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,493 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,493 (Startup thread) - Beginning transaction of type 2
DEBUG 2018-12-14T01:07:42,493 (Startup thread) - Requested query: [UPDATE
jobqueue SET isseed=? WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,494 (Thread-733) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,494 (Thread-733) - Done actual query (0ms): [SET
SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,495 (Thread-734) - Actual query: [UPDATE jobqueue SET
isseed=? WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,495 (Thread-734) - Parameter 0: 'F'
DEBUG 2018-12-14T01:07:42,495 (Thread-734) - Parameter 1: 'S'
DEBUG 2018-12-14T01:07:42,495 (Thread-734) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,496 (Thread-734) - Done actual query (2ms): [UPDATE
jobqueue SET isseed=? WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,496 (Startup thread) - Requested query: [UPDATE
jobqueue SET isseed=?,seedingprocessid=NULL WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,496 (Thread-735) - Actual query: [UPDATE jobqueue SET
isseed=?,seedingprocessid=NULL WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,496 (Thread-735) - Parameter 0: 'S'
DEBUG 2018-12-14T01:07:42,496 (Thread-735) - Parameter 1: 'S'
DEBUG 2018-12-14T01:07:42,497 (Thread-735) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,497 (Thread-735) - Done actual query (1ms): [UPDATE
jobqueue SET isseed=?,seedingprocessid=NULL WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,497 (Startup thread) - Marking for delete for job
1544121003866 all target document references matching 'isnew=B' that are
targets of ( '' )
DEBUG 2018-12-14T01:07:42,497 (Startup thread) - Requested query: [UPDATE
hopcount SET distance=?,deathmark=? WHERE id IN(SELECT ownerid FROM
hopdeletedeps t0 WHERE (t0.jobid=? AND t0.childidhash=?) AND EXISTS(SELECT 'x'
FROM intrinsiclink t1 WHERE (t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND
t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash) AND
t1.isnew=?))]
DEBUG 2018-12-14T01:07:42,497 (Thread-736) - Actual query: [UPDATE hopcount SET
distance=?,deathmark=? WHERE id IN(SELECT ownerid FROM hopdeletedeps t0 WHERE
(t0.jobid=? AND t0.childidhash=?) AND EXISTS(SELECT 'x' FROM intrinsiclink t1
WHERE (t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND
t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash) AND
t1.isnew=?))]
DEBUG 2018-12-14T01:07:42,497 (Thread-736) - Parameter 0: '-1'
DEBUG 2018-12-14T01:07:42,497 (Thread-736) - Parameter 1: 'D'
DEBUG 2018-12-14T01:07:42,497 (Thread-736) - Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,497 (Thread-736) - Parameter 3: ''
DEBUG 2018-12-14T01:07:42,497 (Thread-736) - Parameter 4: 'B'
DEBUG 2018-12-14T01:07:42,498 (Thread-736) - Done actual query (1ms): [UPDATE
hopcount SET distance=?,deathmark=? WHERE id IN(SELECT ownerid FROM
hopdeletedeps t0 WHERE (t0.jobid=? AND t0.childidhash=?) AND EXISTS(SELECT 'x'
FROM intrinsiclink t1 WHERE (t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND
t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash) AND
t1.isnew=?))]
DEBUG 2018-12-14T01:07:42,498 (Startup thread) - Done marking for delete for
job 1544121003866
DEBUG 2018-12-14T01:07:42,498 (Startup thread) - Requested query: [DELETE FROM
intrinsiclink WHERE (jobid=? AND childidhash=?) AND isnew=?]
DEBUG 2018-12-14T01:07:42,499 (Thread-737) - Actual query: [DELETE FROM
intrinsiclink WHERE (jobid=? AND childidhash=?) AND isnew=?]
DEBUG 2018-12-14T01:07:42,499 (Thread-737) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,499 (Thread-737) - Parameter 1: ''
DEBUG 2018-12-14T01:07:42,499 (Thread-737) - Parameter 2: 'B'
DEBUG 2018-12-14T01:07:42,499 (Thread-737) - Done actual query (0ms): [DELETE
FROM intrinsiclink WHERE (jobid=? AND childidhash=?) AND isnew=?]
DEBUG 2018-12-14T01:07:42,499 (Startup thread) - Requested query: [DELETE FROM
hopdeletedeps WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND
deathmark=?))]
DEBUG 2018-12-14T01:07:42,499 (Thread-738) - Actual query: [DELETE FROM
hopdeletedeps WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND
deathmark=?))]
DEBUG 2018-12-14T01:07:42,499 (Thread-738) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,499 (Thread-738) - Parameter 1: 'D'
DEBUG 2018-12-14T01:07:42,500 (Thread-738) - Done actual query (1ms): [DELETE
FROM hopdeletedeps WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND
deathmark=?))]
DEBUG 2018-12-14T01:07:42,500 (Startup thread) - Requested query: [UPDATE
hopcount SET deathmark=? WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,500 (Thread-739) - Actual query: [UPDATE hopcount SET
deathmark=? WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,500 (Thread-739) - Parameter 0: 'Q'
DEBUG 2018-12-14T01:07:42,500 (Thread-739) - Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,501 (Thread-739) - Parameter 2: 'D'
DEBUG 2018-12-14T01:07:42,501 (Thread-739) - Done actual query (1ms): [UPDATE
hopcount SET deathmark=? WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,501 (Startup thread) - Done queueing for
re-evaluation for 1544121003866
DEBUG 2018-12-14T01:07:42,501 (Startup thread) - Requested query: [UPDATE
intrinsiclink SET processid=NULL,isnew=? WHERE (jobid=? AND childidhash=?) AND
isnew IN (?,?)]
DEBUG 2018-12-14T01:07:42,501 (Thread-740) - Actual query: [UPDATE
intrinsiclink SET processid=NULL,isnew=? WHERE (jobid=? AND childidhash=?) AND
isnew IN (?,?)]
DEBUG 2018-12-14T01:07:42,501 (Thread-740) - Parameter 0: 'B'
DEBUG 2018-12-14T01:07:42,501 (Thread-740) - Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,501 (Thread-740) - Parameter 2: ''
DEBUG 2018-12-14T01:07:42,501 (Thread-740) - Parameter 3: 'E'
DEBUG 2018-12-14T01:07:42,501 (Thread-740) - Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,502 (Thread-740) - Done actual query (1ms): [UPDATE
intrinsiclink SET processid=NULL,isnew=? WHERE (jobid=? AND childidhash=?) AND
isnew IN (?,?)]
DEBUG 2018-12-14T01:07:42,502 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,503 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,503 (Startup thread) - Beginning transaction of type 1
DEBUG 2018-12-14T01:07:42,503 (Startup thread) - Requested query: [SELECT
status,connectionname FROM jobs WHERE id=? FOR UPDATE]
DEBUG 2018-12-14T01:07:42,504 (Thread-741) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,504 (Thread-741) - Done actual query (0ms): [SET
SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,504 (Thread-742) - Actual query: [SELECT
status,connectionname FROM jobs WHERE id=? FOR UPDATE]
DEBUG 2018-12-14T01:07:42,504 (Thread-742) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,505 (Thread-742) - Done actual query (1ms): [SELECT
status,connectionname FROM jobs WHERE id=? FOR UPDATE]
DEBUG 2018-12-14T01:07:42,505 (Startup thread) - Requested query: [SELECT
transformationname FROM jobpipelines WHERE (ownerid=? AND transformationname IS
NOT NULL)]
DEBUG 2018-12-14T01:07:42,505 (Thread-743) - Actual query: [SELECT
transformationname FROM jobpipelines WHERE (ownerid=? AND transformationname IS
NOT NULL)]
DEBUG 2018-12-14T01:07:42,505 (Thread-743) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,506 (Thread-743) - Done actual query (1ms): [SELECT
transformationname FROM jobpipelines WHERE (ownerid=? AND transformationname IS
NOT NULL)]
DEBUG 2018-12-14T01:07:42,506 (Startup thread) - Requested query: [SELECT
outputname FROM jobpipelines WHERE (ownerid=? AND outputname IS NOT NULL)]
DEBUG 2018-12-14T01:07:42,506 (Thread-744) - Actual query: [SELECT outputname
FROM jobpipelines WHERE (ownerid=? AND outputname IS NOT NULL)]
DEBUG 2018-12-14T01:07:42,506 (Thread-744) - Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,507 (Thread-744) - Done actual query (1ms): [SELECT
outputname FROM jobpipelines WHERE (ownerid=? AND outputname IS NOT NULL)]
DEBUG 2018-12-14T01:07:42,507 (Startup thread) - Beginning transaction of type 1
DEBUG 2018-12-14T01:07:42,507 (Startup thread) - Requested query: [SELECT
classname FROM outputconnections WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,507 (Thread-745) - Actual query: [SELECT classname
FROM outputconnections WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,507 (Thread-745) - Parameter 0: 'Null'
DEBUG 2018-12-14T01:07:42,508 (Thread-745) - Done actual query (1ms): [SELECT
classname FROM outputconnections WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,508 (Startup thread) - Requested query: [SELECT *
FROM outputconnectors WHERE classname=?]
DEBUG 2018-12-14T01:07:42,508 (Thread-746) - Actual query: [SELECT * FROM
outputconnectors WHERE classname=?]
DEBUG 2018-12-14T01:07:42,508 (Thread-746) - Parameter 0:
'org.apache.manifoldcf.agents.output.nullconnector.NullConnector'
DEBUG 2018-12-14T01:07:42,509 (Thread-746) - Done actual query (1ms): [SELECT *
FROM outputconnectors WHERE classname=?]
DEBUG 2018-12-14T01:07:42,509 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,509 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,509 (Startup thread) - Beginning transaction of type 1
DEBUG 2018-12-14T01:07:42,509 (Startup thread) - Requested query: [SELECT
classname FROM repoconnections WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,509 (Thread-747) - Actual query: [SELECT classname
FROM repoconnections WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,509 (Thread-747) - Parameter 0: 'Web'
DEBUG 2018-12-14T01:07:42,510 (Thread-747) - Done actual query (1ms): [SELECT
classname FROM repoconnections WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,510 (Startup thread) - Requested query: [SELECT *
FROM connectors WHERE classname=?]
DEBUG 2018-12-14T01:07:42,510 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,510 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,510 (Startup thread) - Requested query: [UPDATE jobs
SET
failtime=NULL,seedingversion=?,processid=NULL,failcount=NULL,starttime=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,510 (Thread-748) - Actual query: [UPDATE jobs SET
failtime=NULL,seedingversion=?,processid=NULL,failcount=NULL,starttime=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,510 (Thread-748) - Parameter 0: ''
DEBUG 2018-12-14T01:07:42,510 (Thread-748) - Parameter 1: '1544767662340'
DEBUG 2018-12-14T01:07:42,510 (Thread-748) - Parameter 2: 'A'
DEBUG 2018-12-14T01:07:42,510 (Thread-748) - Parameter 3: '1544121003866'
DEBUG 2018-12-14T01:07:42,511 (Thread-748) - Done actual query (1ms): [UPDATE
jobs SET
failtime=NULL,seedingversion=?,processid=NULL,failcount=NULL,starttime=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,511 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,511 (Startup thread) - Committing transaction!
{code}
> Documents unreachable due to hopcount are not considered unreachable on
> cleanup pass
> ------------------------------------------------------------------------------------
>
> Key: CONNECTORS-1562
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1562
> Project: ManifoldCF
> Issue Type: Bug
> Components: Elastic Search connector, Web connector
> Affects Versions: ManifoldCF 2.11
> Environment: Manifoldcf 2.11
> Elasticsearch 6.3.2
> Web inputconnector
> elastic outputconnecotr
> Job crawls website input and outputs content to elastic
> Reporter: Tim Steenbeke
> Assignee: Karl Wright
> Priority: Critical
> Labels: starter
> Attachments: manifoldcf.log.cleanup, manifoldcf.log.init,
> manifoldcf.log.reduced
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> My documents aren't removed from ElasticSearch index after rerunning the
> changed seeds
> I update my job to change the seedmap and rerun it or use the schedualer to
> keep it runneng even after updating it.
> After the rerun the unreachable documents don't get deleted.
> It only adds doucments when they can be reached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
