On Mon, Jul 23, 2012 at 11:31:41AM -0700, Keith Lofstrom wrote: > Is there any way to tell the search spiders to visit once a day > or once a week, rather than four times per hour? Or send them > "recent changes" lists instead of them repeatedly downloading the > same files? Any other ideas for calming down the web crawlers?
I don't know of a good answer but I could suggest a hack. Keep two copies of your robots.txt file and use a cron job to swap them out for an hour or two in the middle of the night or wherever your slowest traffic period falls. _______________________________________________ Dirvish mailing list [email protected] http://www.dirvish.org/mailman/listinfo/dirvish
