Grant Slater wrote: > Erik Johansson wrote: >> Firefishy unblocked google for now. I'm guessing it's too much trouble >> to be worth it to put the wiki config+robots.txt in SVN. (I did look >> for it before asking) > > I had unblocked googlebot, the load spiked and wiki became unresponsive > again in the evening peak. I have now restricted access to non-peak > hours, which seems to work ok.
Have you tried the Google webmaster tools? You can specify the rate of crawl there. And there are a lot of statistics there that may be useful. Regards, Maarten _______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
