[
https://issues.apache.org/jira/browse/NUTCH-719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729254#action_12729254
]
Steven Denny edited comment on NUTCH-719 at 7/9/09 6:17 AM:
------------------------------------------------------------
I've done some investigation on this.
It looks to me as if queues can get reaped to early. I've put in some debug and
this is what I see:
2009-07-09 04:39:50,704 DEBUG fetcher.Fetcher -
FetchItemQueue::getFetchItemQueue() id=http://125.168.254.20
2009-07-09 04:39:50,704 DEBUG fetcher.Fetcher - Created queue:
http://125.168.254.20
....
2009-07-09 04:39:50,704 DEBUG fetcher.Fetcher - reaping: http://125.168.254.20
.....
2009-07-09 04:39:50,705 DEBUG fetcher.Fetcher - addFetchItem: adding item -
http://www.callidan.com/ma100.htm
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - totalSize++:2
http://125.168.254.20 http://www.callidan.com/ma100.htm queuesize: 1
queuecount: 11
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://61.9.216.193,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://216.184.34.250,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://139.146.150.23,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://203.29.78.68,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://150.101.91.39,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue:
http://209.212.110.211, size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://123.176.112.44,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue:
http://117.104.160.130, size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://196.25.73.205,
size: 0
2009-07-09 04:39:50,884 DEBUG fetcher.Fetcher - * queue: http://202.53.7.145,
size: 0
2009-07-09 04:39:50,884 DEBUG fetcher.Fetcher - * queue: http://202.60.67.145,
size: 1
Note that the queue is created and then immediately reaped, and after totalSize
is incremented, that queue does not appear in the list, even though it
supposedly has the item added to it.
The upshot is that the url is never fetched (as the queue has gone) so
totalSize never = 0, and eventually the abort will happen.
In short I'd say this is a sync issue, but I'm not sure where the best place to
lock would be.
Any comments from the author?
was (Author: stevedenny):
I've done some investigation on this.
It looks to me as if queues can get reaped to early. I've put in some debug and
this is what I see:
2009-07-09 04:39:50,704 DEBUG fetcher.Fetcher -
FetchItemQueue::getFetchItemQueue() id=http://125.168.254.20
2009-07-09 04:39:50,704 DEBUG fetcher.Fetcher - Created queue:
http://125.168.254.20
....
2009-07-09 04:39:50,704 DEBUG fetcher.Fetcher - reaping: http://125.168.254.20
.....
2009-07-09 04:39:50,705 DEBUG fetcher.Fetcher - addFetchItem: adding item -
http://www.callidan.com/ma100.htm
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - totalSize++:2
http://125.168.254.20 http://www.callidan.com/ma100.htm queuesize: 1
queuecount: 11
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://61.9.216.193,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://216.184.34.250,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://139.146.150.23,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://203.29.78.68,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://150.101.91.39,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue:
http://209.212.110.211, size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://123.176.112.44,
size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue:
http://117.104.160.130, size: 0
2009-07-09 04:39:50,883 DEBUG fetcher.Fetcher - * queue: http://196.25.73.205,
size: 0
2009-07-09 04:39:50,884 DEBUG fetcher.Fetcher - * queue: http://202.53.7.145,
size: 0
2009-07-09 04:39:50,884 DEBUG fetcher.Fetcher - * queue: http://202.60.67.145,
size: 1
Not that the queue is created and then immediately reaped, and after totalSize
is incremented, that queue does not appear in the list, even though it
supposedly has the item added to it.
It looks as if when items are fed, there's a posibility of the queue being
reaped before the item is added to the queue. However, totalSize is still
incrememented.
The upshot is that the url is never fetched (as the queue has gone) so
totalSize never = 0, and eventually the abort will happen.
In short I'd say this is a sync issue, but I'm not sure where the best place to
lock would be.
Any comments from the author?
> fetchQueues.totalSize incorrect in Fetcher2
> -------------------------------------------
>
> Key: NUTCH-719
> URL: https://issues.apache.org/jira/browse/NUTCH-719
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.0.0
> Reporter: Julien Nioche
>
> I had a look at the logs generated by Fetcher2 and found cases where there
> were no active fetchQueues but fetchQueues.totalSize was != 0
> fetcher.Fetcher2 - -activeThreads=200, spinWaiting=200,
> fetchQueues.totalSize=1, fetchQueues=0
> since the code relies on fetchQueues.totalSize to determine whether the work
> is finished or not the task is blocked until the abortion mechanism kicks in
> 2009-03-12 09:27:38,977 WARN fetcher.Fetcher2 - Aborting with 200 hung
> threads.
> could that be a synchronisation issue? any ideas?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
