[
https://issues.apache.org/jira/browse/NUTCH-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538725#comment-13538725
]
Tejas Patil edited comment on NUTCH-1284 at 12/22/12 10:54 AM:
---------------------------------------------------------------
I searched for the relevant mail thread[0] to get an idea why this bug was
created.
Quick recap of the issue:
Despite fetcher.max.crawl.delay was set to -1, nutch was marking the url as
ROBOTS_DENIED. With fetcher.max.crawl.delay= -1, the expected behavior is to
wait the amount of time retrieved from robots.txt Crawl-Delay, however long
that might be.
Lewis could reproduce the issue. He suggested the change mentioned in the bug
and hinted that there might be some problem with that property.
An additional condition was needed which prevents urls from being marked
DB_GONE when fetcher.max.crawl.delay= -1 (ie. maxCrawlDelay = -1000). After
this change, I tested with the scenario mentioned in [0] and it worked fine.
[0]:
http://lucene.472066.n3.nabble.com/Re-Re-Re-Re-fetcher-max-crawl-delay-1-doesn-t-work-tc3749639.html
was (Author: tejasp):
I searched for the relevant mail thread[0] to get an idea why this bug was
created.
Quick recap of the issue:
Despite fetcher.max.crawl.delay was set to -1, nutch was marking the url as
ROBOTS_DENIED. With fetcher.max.crawl.delay= -1, the expected behavior is to
wait the amount of time retrieved from robots.txt Crawl-Delay, however long
that might be.
Lewis could reproduce the issue. He suggested the change mentioned in the bug
and hinted that there might be some problem with that property.
An additional condition was needed to be changed which prevents urls from being
marked DB_GONE when fetcher.max.crawl.delay= -1 (ie. maxCrawlDelay = -1000).
After this change, I tested with the scenario mentioned in [0] and it worked
fine.
[0]:
http://lucene.472066.n3.nabble.com/Re-Re-Re-Re-fetcher-max-crawl-delay-1-doesn-t-work-tc3749639.html
> Add site fetcher.max.crawl.delay as log output by default.
> ----------------------------------------------------------
>
> Key: NUTCH-1284
> URL: https://issues.apache.org/jira/browse/NUTCH-1284
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: nutchgora, 1.5
> Reporter: Lewis John McGibbney
> Priority: Trivial
> Fix For: 1.7
>
> Attachments: NUTCH-1284.patch
>
>
> Currently, when manually scanning our log output we cannot infer which pages
> are governed by a crawl delay between successive fetch attempts of any given
> page within the site. The value should be made available as something like:
> {code}
> 2012-02-19 12:33:33,031 INFO fetcher.Fetcher - fetching
> http://nutch.apache.org/ (crawl.delay=XXXms)
> {code}
> This way we can easily and quickly determine whether the fetcher is having to
> use this functionality or not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
