Maarten Deen wrote: > Grant Slater wrote: > >> Erik Johansson wrote: >> >>> Firefishy unblocked google for now. I'm guessing it's too much trouble >>> to be worth it to put the wiki config+robots.txt in SVN. (I did look >>> for it before asking) >>> >> I had unblocked googlebot, the load spiked and wiki became unresponsive >> again in the evening peak. I have now restricted access to non-peak >> hours, which seems to work ok. >> > > Have you tried the Google webmaster tools? You can specify the rate of crawl > there. And there are a lot of statistics there that may be useful. >
Yup. I tried tweaking it awhile back. Still had problems. Sitemap did help. http://wiki.openstreetmap.org/sitemap-index-wiki.xml The server is only an Intel Atom 230. It runs Wiki, SVN, All outbound email and Lists. / Grant _______________________________________________ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev