[
https://issues.apache.org/jira/browse/NUTCH-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12578742#action_12578742
]
Andrzej Bialecki commented on NUTCH-615:
-----------------------------------------
I think the code in ParseOutputFormat doesn't matter that much. Any
CrawlDatum-s created with LINKED status will be used only as a source of
metadata in CrawlDbReducer, and if it defines a truly new URL then the
FetchSchedule will be initialized in CrawlDbReducer anyway.
So, I think we could apply the parts of the patch in Fetcher-s, and skip the
ParseOutputFormat part.
> Redirected URL are fetched wihtout setting any FetchInterval
> ------------------------------------------------------------
>
> Key: NUTCH-615
> URL: https://issues.apache.org/jira/browse/NUTCH-615
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.0.0
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
> Attachments: NUTCH-615.patch, NUTCH-615_v2.patch
>
>
> An url which is redirected result to a new URL. We create a new CrawlDatum
> for the new URL within the Fetcher but the FetchInterval was not initialized.
> The new url was recorded in the DB with a FetchInterval = 0 and the FetchTime
> is never correctly updated to be fetch later in the future. Thus we keep
> crawling those URL at each generation.
> This patch fix this issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
