You can add a "Crawl-delay: xx" instruction, where "xx" is a delay value between successive crawler accesses [http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html]
If I do this for Nutch bot, will it work? Does it get Crawl-delay instructions and update the interval? Any other similar ones for threads.... -- View this message in context: http://www.nabble.com/crawl-delay-and-nutch-tf3528305.html#a9845743 Sent from the Nutch - User mailing list archive at Nabble.com. ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
