[ http://issues.apache.org/jira/browse/NUTCH-293?page=comments#action_12421930 ] Sami Siren commented on NUTCH-293: ----------------------------------
perhaps instead of delay = crawlDelay > 0 ? crawlDelay : serverDelay; we could do delay=Math.max(crawlDelay, serverDelay); also the delay could be calculated only once and passed as a parameter to blockAddr, unblockAddr > support for Crawl-delay in Robots.txt > ------------------------------------- > > Key: NUTCH-293 > URL: http://issues.apache.org/jira/browse/NUTCH-293 > Project: Nutch > Issue Type: Improvement > Components: fetcher > Affects Versions: 0.8-dev > Reporter: Stefan Groschupf > Priority: Critical > Attachments: crawlDelayv1.patch > > > Nutch need support for Crawl-delay defined in robots.txt, it is not a > standard but a de-facto standard. > See: > http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html > Webmasters start blocking nutch since we do not support it. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
