On Wed, Jul 10, 2013 at 9:36 AM, Magnus Hagander <[email protected]> wrote: > We already run this, that's what we did to make it survive at all. The > problem is there are so many thousands of different URLs you can get > to on that site, and google indexes them all by default.
There's also https://support.google.com/webmasters/answer/48620?hl=en which lets us control how fast the Google crawler crawls. I think it's adaptive though so if the pages are slow it should be crawling slowly -- greg -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
