On Thu, Feb 23, 2012 at 2:07 PM, Baksik, Corinna M. <corinna_bak...@harvard.edu> wrote: > We don't allow crawlers because it has caused serious performance issues in > the past.
There are simple solutions that may help for the performance problem. You can use Crawl-delay: http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive You can set Google's crawl rate in Webmaster Tools as well. Jason
