On Mon, Apr 20, 2015 at 7:23 AM, Barry Hunter <barrybhun...@gmail.com> wrote:
> The Ahref bot (if its the legitimate one of course!) definitly obays the > robots.txt > https://ahrefs.com/robot > > Looking at http://www.myhotelcar.com/robots.txt there is nothing blocking > that particular bot. > +1. You can also try looking into Cloudflare <https://www.cloudflare.com/google> to proxy your site and filter out some robots. More importantly: are the robots hitting all of your pages, or are they only hitting certain types of pages? Are they perhaps repeatedly retrieving a RSS feed or sitemap documents? ----------------- -Vinny P Technology & Media Consultant Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CALSvALD1ezGi5V1myez50nSCGxjLCwTPrTnK3LuThOGM%2Bd9nEA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
