On Wed, Feb 8, 2012 at 2:55 AM, Pandu Poluan <pa...@poluan.info> wrote: > > On Jan 27, 2012 11:18 PM, "Paul Hartman" <paul.hartman+gen...@gmail.com> > wrote: >> > > ---- >8 snippage > >> >> BTW, the Baidu spider hits my site more than all of the others combined... >> > > Somewhat anecdotal, and definitely veering way off-topic, but Baidu was the > reason why my company decided to change our webhosting company: Its > spidering brought our previous webhosting to its knees... > > Rgds,
I wonder if Baidu crawler honors the Crawl-delay directive in robots.txt? Or I wonder if Baidu cralwer IPs need to be covered by firewall tarpit rules. ;)
