Thanks Vinny, I think you have picked the issue correctly they are hitting particular set of pages regularly hotel pages which were dynamically generated, you are correct about rss and sitemap feed. So please tell me the way to overcome this issue as these spam bots specially ahref bot is consuming my server bandwidth a lot un-necessarily. I want a good solution so that I will not face any spam bot hurdle in future.
On Tuesday, April 21, 2015 at 9:49:30 AM UTC+5:30, Vinny P wrote: > > On Mon, Apr 20, 2015 at 7:23 AM, Barry Hunter <barryb...@gmail.com > <javascript:>> wrote: > >> The Ahref bot (if its the legitimate one of course!) definitly obays the >> robots.txt >> https://ahrefs.com/robot >> >> Looking at http://www.myhotelcar.com/robots.txt there is nothing >> blocking that particular bot. >> > > > +1. > > You can also try looking into Cloudflare > <https://www.cloudflare.com/google> to proxy your site and filter out > some robots. > > More importantly: are the robots hitting all of your pages, or are they > only hitting certain types of pages? Are they perhaps repeatedly retrieving > a RSS feed or sitemap documents? > > > ----------------- > -Vinny P > Technology & Media Consultant > Chicago, IL > > App Engine Code Samples: http://www.learntogoogleit.com > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/da8dce4a-d677-4f73-b39b-31576cbb31fa%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
