On Tue, Jan 19, 2010 at 09:15:17AM -0500, David Golden wrote: > On Tue, Jan 19, 2010 at 8:50 AM, Barbie <bar...@missbarbell.co.uk> wrote: > > I have had an apology from the Bing representative, and I've said that I > > will only remove the ban once I see evidence that the bot is being more > > responsible. I have only banned it from www.cpantesters.org, all the > > other sites are unaffected. The reduction in HTTP traffic though has > > been dramatic. > > Does redirecting the MSN bot to the static site help at all? (I > though the issue was that spidering the site triggers page > regeneration.)
It can do, but that wasn't the problem. It was trying to read lots of dynamically created pages all at once. They are able to index the static site already, as that's at static.cpantesters.org, a completely different domain. And having just checked indeed they are indexing it, albeit at a considerably slower speed than the www.cpantesters.org domain. After the last big hit a lot of the dynamic stuff is now pre-built by the Builder, such that now the actual dynamic parts are mostly pushed into the client-side javascript. It's the fact that the site changes rapidly and the bot tries to grab as much as possible as quickly as it can. Cheers, Barbie. -- Birmingham Perl Mongers <http://birmingham.pm.org> Memoirs Of A Roadie <http://barbie.missbarbell.co.uk> CPAN Testers Blog <http://blog.cpantesters.org> YAPC Conference Surveys <http://yapc-surveys.org>
