On Wed, Jul 10, 2013 at 9:36 AM, Magnus Hagander <mag...@hagander.net> wrote: > We already run this, that's what we did to make it survive at all. The > problem is there are so many thousands of different URLs you can get > to on that site, and google indexes them all by default.
There's also https://support.google.com/webmasters/answer/48620?hl=en which lets us control how fast the Google crawler crawls. I think it's adaptive though so if the pages are slow it should be crawling slowly -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
